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DESCRIPTION 



METHOD, APPARATUS, AND PROGRAM FOR PROCESSING AN IMAGE 



Technical Field 



5 



The present invention relates to a method, apparatus 



and program for processing an image, and more particularly, 
to a method, apparatus and program for processing an image 
to obtain a sharp image using an imaging device including a 
single-plane sensor . 

10 

Background Art 

In recent years, a digital camera has become very 
popular as a camera. In the digital camera, an image is 
taken using an imaging device such as a CCD (Charge Coupled 

15 Device) or a CMOS (Complementary Metal Oxide Semiconductor) 
and the image is displayed on a monitor such as a LCD 
(Liquid Crystal Display) disposed on the digital camera such 
that a user can view the image (hereinafter, such an image 
taken by the imaging device will be referred to simply as a 

2 0 captured image) . An image signal of the captured image is 
converted into digital form and transferred to an image 
processing apparatus such as a personal computer (PC) via a 
storage medium such as a flash memory or via wired or 
wireless communication using a cable or an infrared ray. On 

25 receiving the image data captured by the digital camera, the 
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personal computer displays the image on a monitor such as a 
CRT (Cathode Ray Tube) or a LCD (Liquid Crystal Display) so 
that a user can view the image and edit it. 

When an image of a subject that is not well lighted is 
5 taken by a digital camera in a darkish environment such as a 
shaded place or a poorly lighted room, the shutter speed 
should be set to be low enough to obtain sufficient exposure 
(that is, the exposure time should be long enough) . 

When the exposure time is long, if the digital camera 

10 is firmly held, for example, by placing it on a tripod such 
that the digital camera does not shake, a good image can be 
obtained. However, in a case in which the digital camera is 
held in hands, the digital camera can shake. If the camera 
shakes when the shutter is in an open state (during a 

15 shutter time) , a resultant captured image of a subject is 

blurred. The blurred image is called a "hand shake blurred" 
image or a "camera shake blurred" image. 

It is known to obtain equivalently sufficient exposure 
by simply cumulatively superimposing two or more captured 

20 images, instead of decreasing the shutter speed (increasing 
the shutter time) (an example of such a technique may be 
found, for example, in Japanese Unexamined Patent 
Application Publication No. 05-236422) . 

However, in the technique disclosed in Japanese 

25 Unexamined Patent Application Publication No. 05-236422, 
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because two or more captured images are simply superimposed, 
an image is blurred if the digital camera shakes, as in the 
case in which an image is captured at a low shutter speed. 

In some digital cameras available from Canon Kabushiki 
5 Kaisha, a technique called an image stabilizer (IS) is used 
to prevent an image from being blurred when a digital camera 
shakes . 

In the technique using the image stabilizer, a shake or 
vibration of a digital camera is detected by a pre-sensor 

10 disposed in an optical lens system, and a part (correction 
optical system) of the lens system is moved in a direction 
perpendicular to the optical axis in accordance with a 
digital signal indicating the detected shape or vibration of 
the digital camera so that refraction of light is changed to 

15 prevent the image from being blurred. 

The^ image stabilizer makes it possible to cancel out 
blurring of an image due to shaking of a camera or a stage 
on which the camera is placed caused by shaking of a hand or 
a wind, which is significant in particular when an image is 

20 taken at a low shutter speed by a digital camera held in 
hands, thereby allowing a user to obtain a sharp image. 

However, the image stabilizer needs a dedicated sensor 
responsible for sensing shaking of the camera and also needs 
a mechanism for quickly moving a part (correction optical 

25 system) of the lens system, and thus the image stabilizer 
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has a complicated structure, which results in an increase in 
production cost. 

Another known technique to prevent an image from being 
blurred when the image is captured by a shaking camera is to 
5 successively capture a plurality of images at a high shutter 
speed, detect the displacement of second and following 
captured images relative to a first captured image, 
compensate for the displacement of the second and following 
captured images, and superimposing the compensated images on 

10 the first captured image (examples of this technique may be 
found, for example, in Japanese Unexamined Patent 
Application Publications Nos. 2000-217032, 2000-224460, 
2000-244803, 2000-244797, 2000-069352, 10-341367, and 09- 
261526) . Basically, in this conventional technique, images 

15 similar to a first captured image are produced from second 
and following captured images by means of correction or 
interpolation, and the resultant images are simply added or 
linearly added on a pixel-by-pixel basis to the first 
captured image . 

20 In this technique, each of images successively captured 

at a high shutter speed has practically no blur although 
images are dark. An image finally obtained by superimposing 
of second and following captured images on a first image can 
have high brightness similar to that obtained when the 

25 exposure time is set properly. 
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In this technique, the correction of the detected 
displacement of the second and following captured images 
relative to the first captured image is made by 
interpolating color signals (data) including a R (Red) 
5 signal (red data) , a G (Green) signal (green data) , and a B 
(Blue) signal (blue data) for each pixel, by using an 
interpolation function based on linear interpolation, 
bicubit interpolation, or the like. 

However, in the case of an imaging device using a 

10 single-plane sensor as an imaging device, each pixel outputs 
only one of color signals, that is, an R signal, a G signal, 
or a B signal. This makes it impossible to correct 
displacements of a plurality of images by means of 
interpolation described above, and any interpolation method 

15 applicable to the single-plane sensor is known. 

Disclosure of Invention 

In view of the above, it is an object of the present 
invention to provide a technique of obtaining a sharp image 
2 0 captured by, for example, an imaging device using a single- 
plane sensor even in a state in which camera shake occurs. 

The present invention provides an image processing 
method, comprising the steps of detecting a positional 
relationship among a plurality of input images captured by 
25 image capture means for capturing an image each pixel of 
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which has one of n pixel values, and estimating an output 
image by identifying observed pixel components that should 
be taken into consideration from the plurality of input 
images for each pixel position of the output image, and 
5 producing the output image having (n + 1) or more pixel 

values for each pixel based on the identified observed pixel 
components . 

In the step of estimating the output image, observed 
pixel components that should be taken into consideration may 

10 be selected from the plurality of input images for each 
pixel position of the output image, and the output image 
having (n + 1) or more pixel values for each pixel may be 
produced based on the selected observed pixel components. 

In the step of estimating the output image, the output 

15 image having (n + 1) or more color signals for each pixel 

may be produced from the plurality of input images, based on 
the positional relationship detected in the detection step. 

In the step of estimating the output image, the output 
image may be determined taking into account a color 

20 correlation at each pixel position and/or in the vicinity of 
each pixel position of the output image. In the step of 
estimating the output image, pixel values of pixels of the 
input images may be changed to pixel values depending on the 
degree of steepness of an edge, and the output image may be 

25 estimated based on the resultant changed pixel values. 
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n may be equal to 1, and in the step of estimating the 
output image, the output image having three pixel values for 
each pixel may be produced from the plurality of input 
images, based on the positional relationship detected in the 
5 detection step. 

The image capture means may be a single -plane sensor, 
in which a color signal observed varies depending on the 
position of each input image. 

The image capture means may capture the plurality of 
10 input images with underexposure, and the image processing 
method may further comprise the step of correcting the 
exposure by gaining up each pixel value of the plurality of 
input images . 

The step of estimating the output image may comprise 
15 the steps of determining a corrected position of each pixel 
depending on the positional relationship detected in the 
detection step for each pixel of each of the plurality of 
input images, producing a light intensity integral 
constraint formula indicating that the difference between 
20 the integral of light intensity in a region centered at the 
corrected position and a pixel value of a pixel at the 
corrected position should be less than a predetermined 
error; and calculating the light intensity integral 
constraint formula . 
25 When the light intensity integral constraint formula is 
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produced, if the plurality of input images have been gained 
up, the error may be changed depending on the factor of the 
gaining up. 

In the step of estimating the output image, a corrected 
5 position of each pixel may be determined depending on the 
positional relationship detected in the detection step for 
each pixel of each of the plurality of input images, and the 
output image may be determined taking into account a pixel 
value of a pixel of the input images whose corrected 

10 position is located in the vicinity of each pixel position 
of the output image and also taking into account the error 
depending on the factor of the gaining up. 

In the step of estimating the output image, a corrected 
position of each pixel may be determined depending on the 

15 positional relationship detected in the detection step for 

each pixel of each of the plurality of input images, and the 
output image may be determined taking into account a pixel 
value of a pixel of the input images whose corrected 
position is located in the vicinity of each pixel position 

2 0 of the output image and also taking into account the 

distance between the corrected position and the position of 
the corresponding pixel of the output image . 

In the step of estimating the output image, a corrected 
position of each pixel may be determined depending on the 

25 positional relationship detected in the detection step for 
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each pixel of each of the plurality of input images, and the 
output image may be determined taking into account a pixel 
value of a pixel of the input images whose corrected 
position is located in the vicinity of each pixel of the 
5 output image and also taking into account the error 
depending on the factor of the gaining up. 

In the step of estimating the output image, a corrected 
position of each pixel may be determined depending on the 
positional relationship detected in the detection step for 

10 each pixel of each of the plurality of input images, and the 
output image may be estimated by determining the weighted 
sum of pixel values of the input images whose corrected 
position is located in the vicinity of each pixel position 
of the output image . 

15 In the step of estimating the output image, the correct 

pixel value of a pixel of interest may be determined by 
determining the weighted sum of pixel values of the input 
images whose corrected position is located in the vicinity 
of each pixel position of the output image, weights of the 

2 0 weighted sum being determined using a function having a 

lowpass-f ilter characteristic with respect to the distance 
between the position of the pixel of interest in the output 
image and the corrected position. 

In the step of estimating the output image, the correct 

25 pixel value of the pixel of interest may be estimated by 
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selectively performing a first calculation process of 
determining the weighted sum of pixel values of the input 
images whose corrected position is located in the vicinity 
of the position of the pixel of interest, or a second 
5 calculation process of determining the weighted sum of pixel 
values of the input images whose corrected position is 
located in the vicinity of the position of a pixel of the 
output image located adjacent to the pixel of interest. 

In the step of estimating the output image, the second 
10 calculation process may be selected when the sum of weights 
in the first calculation process is equal or nearly equal to 
0 . 

In the step of estimating the output image, the correct 
pixel value of the pixel of interest may be determined by 

15 selectively performing a first calculation process of 

determining the weighted sum of pixel values of the input 
images whose corrected position is located in the vicinity 
of the position of the pixel of interest, or a second 
calculation process of determining the weighted sum of pixel 

20 values of the input images whose corrected position is 

located in a second vicinity of the position of the pixel of 
interest, the second vicinity being greater than the first 
vicinity . 

In the step of estimating the output image, the second 
25 calculation process may be selected when the sum of weights 
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in the first calculation process is equal or nearly equal to 
0 . 

In the step of estimating the output image, the correct 
pixel value of the pixel of interest may be determined by 
5 selectively performing a first calculation process of 
determining the sum of pixel values weighted by a first 
weighting factor of the input images whose corrected 
position is located in the vicinity of the position of the 
pixel of interest, or a second calculation process of 

10 determining the sum of pixel values weighted by a second 
weighting factor of the input images whose corrected 
position is located in the vicinity of the position of the 
pixel of interest. 

In the step of estimating the output image, the second 

15 calculation process may be selected when the sum of weights 
in the first calculation process is equal or nearly equal to 
0 . 

In the detection step, the positional relationship 
among the plurality of input images may be detected with 
2 0 reference to an input image captured at the exact or nearly 
exact center of a time during which the plurality of input 
images were captured. 

The present invention provides an image processing 
apparatus comprising image capture means for capturing a 
25 plurality of input images having n pixel values for each 



- 12 - 

S04P0831 

pixel; detection means for detecting a positional 
relationship among the plurality of input images, and means 
for estimating an output image by identifying observed pixel 
components that should be taken into consideration from the 
5 plurality of input images for each pixel position of the 

output image, and producing the output image having (n + 1) 
or more pixel values for each pixel based on the identified 
observed pixel components. 

The present invention provides a program for causing a 

10 computer to execute a process comprising the steps of 

detecting a positional relationship among a plurality of 
input images captured by image capture means for capturing 
an image each pixel of which has one of n pixel values, and 
estimating an output image by identifying observed pixel 

15 components that should be taken into consideration from the 
plurality of input images for each pixel position of the 
output image, and producing the output image having (n + 1) 
or more pixel values for each pixel based on the identified 
observed pixel components. 

2 0 According to the present invention, a positional 

relationship among a plurality of input images having n 
pixel values at each pixel is detected. Of observed pixel 
components of the plurality of input images, observed pixel 
components that should be taken into consideration are 

25 identified for each pixel position of an output image, based 
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on the detected positional relationship. Based on the 
observed pixel components identified, an output image having 
(n + 1) or more pixel values at each pixel is . estimated . 

The image processing apparatus may be in the form of a 
5 self-contained apparatus or a block responsible for image 
processing in an apparatus. 



Brief Description of the Drawings 

Fig. 1 is a block diagram showing an example of a 
10 configuration of a digital camera 1 according to an 
embodiment of the present invention. 

Fig. 2 is a flow chart of a process performed by the 
digital camera 1 shown in Fig. 1 to take an image. 

Fig. 3 is a diagram showing a manner in which pixels of 
15 an imaging device 4 shown in Fig. 1 are arranged. 

Fig. 4 is a block diagram showing an example of 
detailed configuration of a signal processor 7 shown in Fig. 
1 . 

Fig. 5 is a diagram showing a relationship in terms of 
20 coordinates between a first image and a second image. 

Fig. 6 a flow chart showing an image estimation process 
according to a first embodiment. 

Fig. 7 a flow chart showing an image estimation process 
according to the first embodiment. 
25 Fig. 8 is a diagram showing a color correlation 
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constraint according to a second embodiment . 

Fig. 9 is a diagram showing a color correlation 
constraint according to a second embodiment . 

Fig. 10 a flow chart showing an image estimation 
process according to a second embodiment. 

Fig. 11 a flow chart showing an image estimation 
process according to the second embodiment. 

Fig. 12 a flow chart showing an image estimation 
process according to the second embodiment. 

Fig. 13 is a diagram showing a first image stored in a 
f r ame memory 22-1. 

Fig. 14 is a diagram showing a second image stored in a 
frame memory 22-2. 

Fig. 15 is a diagram showing a third image stored in a 
frame memory 22-3. 

Fig. 16 is a diagram showing pixels of the imaging 
device 4 . 

Fig. 17 is a diagram showing a manner in which pixels 
of the imaging device 4 shown in Fig. 1 are arranged. 
Fig. 18 is a diagram showing a spring model. 
Fig. 19 is a diagram showing a spring model. 
Fig. 20 is a diagram showing a spring model. 
Fig. 21 is a diagram showing a spring model. 
Fig. 22 is a diagram showing a spring model. 
Fig. 23 is a diagram showing a spring model. 
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Fig. 24 a flow chart showing an image estimation 
process according to a third embodiment. 

Fig. 25 a flow chart showing an image estimation 
process according to a fourth embodiment. 
5 Fig. 26 a flow chart showing a correct image estimation 

process according to a fifth embodiment. 

Fig. 27 is a diagram showing a state in which green 
components (G signals) are sensed by respective pixels of 
the imaging device 4 . 
10 Fig. 28 is a diagram showing an edge extending in a 

vertical direction . 

Fig. 2 9 is a diagram showing an edge extending in a 
horizontal direction . 

Fig. 3 0 is a diagram showing an edge extending in a 
15 direction from upper left to lower right. 

Fig. 31 is a diagram showing an edge extending in a 
direction from upper right to lower left. 

Fig. 32 is a diagram showing a plane Ql produced when 
an edge extending in the vertical direction is detected. 
2 0 Fig. 3 3 is a diagram showing a plane Q2 produced when 

an edge extending in the horizontal direction is detected. 

Fig. 34 is a diagram showing a plane Q3 produced when 
an edge extending in the direction from upper left to lower 
right . 

25 Fig. 35 is a diagram showing a plane Q4 produced when 
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an edge extending in the direction from upper right to lower 
left. 

Fig. 3 6 a flow chart showing an image estimation 
process according to a sixth embodiment. 

Fig. 37 is a diagram showing a color correlation 
constraint according to a seventh embodiment . 

Fig. 38 is a diagram showing a color correlation 
constraint according to the seventh embodiment. 

Fig. 3 9 a flow chart showing an image estimation 
process according to the seventh embodiment. 

Fig. 4 0 is a diagram showing the difference between the 
present invention and a conventional technique. 

Fig. 41 is a block diagram showing an example of a 
configuration of a computer according to an embodiment of 
the present invention. 

Fig. 42 is a diagram showing a cubic function. 

Fig. 43 is a diagram showing pixel values at a position 
(I 1 , J 1 ) in a reference coordinate system. 

Fig. 44 is a diagram showing pixel values in an 
exceptional state at a position (I 1 , J') in the reference 
coordinate system . 

Fig. 45 is a diagram showing an exception handling 
process for a G signal. 

Fig. 46 is a diagram showing an exception handling 
process for a G signal. 
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Fig. 47 is a diagram showing an exception handling 
process for a R signal. 

Fig. 48 is a diagram showing an exception handling 
process for a R signal. 
5 Fig. 49 a flow chart showing an image estimation 

process according to an eighth embodiment. 

Fig. 50 a flow chart showing an image estimation 
process according to the eighth embodiment. 

Fig. 51 a flow chart showing an image estimation 
10 process according to the eighth embodiment. 

Fig. 52 a flow chart showing an image estimation 
process according to the eighth embodiment. 

Fig. 53 a flow chart showing an image estimation 
process according to the eighth embodiment. 
15 Fig. 54 is a diagram showing captured images 4 01i to 

401 8 . 

Fig. 55 is a diagram showing an output image produced 
from captured images by using a first captured image as a 
reference image . 

20 Fig. 56 is a diagram showing an output image produced 

from captured images by using an intermediate captured image 
as a reference image . 

Best Mode for Carrying Out the Invention 
25 Fig. 1 is a block diagram showing an example of a 
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configuration of a digital (still) camera 1 according to an 
embodiment of the present invention. 

The digital camera 1 shown in Fig. 1 includes a lens 2, 
an aperture 3 , an imaging device 4 , a correlated double 
5 sampler 5, a A/D (Analog/Digital) computer 6, a signal 

processor 7, a timing generator 8, a D/A (Digital/Analog) 
converter 9, a video encoder 10, a monitor 11, a codec 12, a 
memory 13, a bus 14, a CPU (Central Processing Unit) 15, and 
an input device 16. The A/D converter 6 includes a shifter 

10 12, and the signal processor 7 includes a frame memory 22. 

Light originating from a subject (not shown) passes 
through an optical system including the lens 2 and the 
aperture 3, and is incident on the imaging device 4. The 
imaging device 4 is in the form of a single-plane sensor 

15 made of a CCD or a CMOS including a particular number of 
pixels (picture elements) . 

The imaging device 4 senses the incident light 
originating from the subject for a specified period (shutter 
time) at specified intervals in accordance with an exposure 

20 timing signal supplied from the timing generator 8. The 

imaging device 4 converts the intensity of light sensed by 
each of photosensor elements disposed in an image sensing 
plane to an electric signal by means of photoelectric 
conversion. The resultant electric signal is supplied as an 

25 image signal to the correlated double sampler 5. Because 
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the imaging device 4 is constructed in the form of the 
single-plane sensor, each pixel of the image supplied in the 
form of the electric image signal to the correlated double 
sampler 5 is one of R, G, and B signals (data) . 
5 The imaging device 4 captures a plurality of images (N 

images) at a higher shutter speed (with a longer shutter 
time) than a shutter speed (shutter timer (exposure time) ) 
necessary to obtain optimum exposure for one image, for use 
to produce a sharp output image even if a camera shake 

10 occurs. Each of N images (input images) captured (with 
underexposure) by the imaging device 4 is darker than is 
obtained if the image is captured with optimum exposure. 
That is, the brightness of each image is lower by a factor 
of Mk (k = 1 to N) than that obtained with optimum exposure 

15 (that is, equal to l/Mk times that obtained with optimum 
exposure) . The value of Mk is determined by the shutter 
speed . 

The correlated double sampler 5 removes noise from the 
image signal (electric signal) received from the imaging 

20 device 4 by means of correlated double sampling and supplies 
the resultant image signal to the A/D converter 6 . On 
receiving the noise-removed image signal of the subject from 
the correlated double sampler 5, the A/D converter 6 
converts it into a digital signal, that is, the A/D 

25 converter 6 samples the received image signal and quantizes 
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it. Thereafter, the shifter 21 increases (gains up) the 
magnitude of the digital image signal representing a dark 
image captured with underexposure by a factor of Mk, for 
example, by means of n ! -bit shifting so as to obtain an 
5 image with brightness (signal level) similar to that 

obtained with optimum exposure. The resultant image signal 
is supplied to the signal processor 7. 

Note that the correlated double sampler 5 cannot remove 
all noise components in the above-described process of 

10 removing noise from the image signal, and thus the image 

signal output from the correlated double sampler 5 includes 
a residual noise component. The residual noise component 
remaining without being removed by the correlated double 
sampler 5 causes an error for the true value of the image 

15 signal, and the error is also increased by the shifter 21 by 
a factor of Mk as well as the image signal. Thus, the 
resultant error depends on the factor of the gaining up 
performed by the shifter 21. Let E denote the magnitude of 
the residual noise component including the image signal 

20 output from the correlated double sampler 5. The noise E is 
set to a maximum possible value depending on the 
characteristic of the imaging device 4 . The image signal 
supplied from the A/D converter 6 to the signal processor 7 
includes noise with a magnitude approximately equal to Mk 

25 times the noise E (Ex Mk) . For example, when Mk = 8 , n ' = 
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set to be 3, and the shifter 21 performs 3 -bit shifting on 
the k-th captured image to increase the brightness of the k- 
th captured image to proper brightness. 

After the image signal is gained up by the factor of Mk 
5 such that the brightness is increased to the optimum level, 
the image signal of each of N captured images is output from 
the A/D converter 6 and temporarily stored in the frame 
memory 22 of the signal processor 7. 

In accordance with programs prepared in advance, the 

10 signal processor 7 performs a process on the image signal of 
each of N captured images with brightness gained up to the 
proper level stored in the frame memory 22. 

More specifically, the signal processor 7 employs the 
first one of the N captured image as a reference image, and 

15 sequentially selects second and N-th captured image as a 

target image. The signal processor 7 detects the positional 
displacement of the target image relative to the reference 
image. The signal processor 7 compensates for the camera 
shake based on the detected positional displacement and 

20 produces a displacement-compensated sharp image (to be 

output) having all G, R, and B signals for each pixel. The 
resultant output image signal is supplied to one or both of 
the D/A converter 9 and codec 12 . The signal processor 7 
may be realized using a DSP (Digital Signal Processor) or 

25 the like. In the following discussion, it is assumed that 
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the image signal processed by parts following the A/D 
converter 6 has been gained up in brightness to a level 
similar to that obtained with optimum exposure. 

The timing generator 8 supplies the exposure timing 
5 signal to the imaging device 4, the correlated double 

sampler 5, the A/D converter 6, and the signal processor 7 
such that N images are captured at the specified intervals. 
An use may change the interval, for example, depending on 
the brightness of a subject. If the interval is changed by 

10 the user via the input device 16, the data indicating the 

changed value of the interval is supplied from the CPU 15 to 
the timing generator 8 via the bus 14 . 

The D/A converter 9 converts the image signal output 
from the signal processor 7 into an analog signal and 

15 supplies the resultant analog image signal to the video 

encoder 10. The video encoder 10 converts the image signal 
(in the form of the analog signal) supplied from the D/A 
converter 9 into a video signal in a form capable of being 
displayed on the monitor 11. The resultant video signal is 

20 supplied to the monitor 11. The monitor 11 serves as a 

viewfinder of the digital camera 1, and it may be realized 
using a LCD or a CRT. The video signal supplied from the 
video encoder 10 is displayed on the monitor 11. Thus, a 
sharp image is displayed on the monitor 11. 

25 The codec 12 encodes the image signal output from the 



- 23 - 

S04P0831 

signal processor 7 in according to a particular coding 
technique such as that according to the JPEG (Joint 
Photographic Exerts Group) standard, the MPEG (Moving 
Picture Experts Group) standard, or the DV (Digital Video) 
5 standard. The resultant encoded image signal is supplied to 
the memory 13 . 

The memory 13 is made of a semiconductor memory such as 
a flash memory and servers to temporarily or permanently 
store the encoded image signal supplied from the codec 12. 

10 Instead of the memory 13, a storage medium such as a 

magnetic disk or an optical (magnetooptical) disk may be 
used. The memory 13 or the storage medium used instead of 
the memory 13 may be removable from the digital camera 1. 
The digital camera 1 may include both a storage medium 

15 disposed in a fixed fashion in the digital camera 1 and a 
storage medium disposed in a removable fashion on the 
digital camera 1 . 

The CPU 15 supplies a control signal to various parts 
via the bus 14 to control various processes. For example, 

20 in accordance with an image capture start signal issued by a 
user via the input device 16, the CPU 15 supplies the 
control signal to various parts to capture an image of a 
subject and store the captured image in the memory 13. 

The input device 16 includes operation buttons such as 

25 a release button disposed on the main part of the digital 
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camera 1. If an operation button is operated by a user, a 
signal is generated and supplied to the CPU 15 via the bus 
14 . In accordance with the signal received from the input 
device 16 via the bus 14, the CPU 15 controls various parts. 
5 One or more of the operation buttons of the input device 16 
may be displayed on the monitor 11. The operation imposed 
on an operation button displayed on the monitor 11 may be 
detected by a transparent tablet disposed on the monitor 11. 
Referring to a flow chart shown in Fig. 2, a process of 

10 taking an image performed by the digital camera 1 is 
described below. 

First, in step SI, the imaging device 4 takes an image 
of a subject. More specifically, each time the release 
button (shutter button) is pressed, the imaging device 4 

15 successively captures N images at a high shutter speed by 
sensing incident light originating from the subject in 
accordance with the exposure timing signal supplied from the 
timing generator 8. Thus, each time an image is taken, dark 
N captured images with a brightness level lower than that 

20 obtained with optimum exposure are obtained. The light 

originating from the subject is sensed and converted into an 
electric signal. The resultant image signal in the form of 
the electric signal is supplied to the correlated double 
sampler 5. The correlated double sampler 5 removes a noise 

25 component from the image signal and supplies the resultant 
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image signal to the A/D converter 6. Thereafter, the 
process proceeds to step S2 . 

In step S2 , the D/A converter 6 converts the noise- 
removed image signal of the subject received from the 
5 correlated double sampler 5 into a digital signal. 

Furthermore, the shifter 21 converts the underexposed dark 
image into an image with brightness (signal level) similar 
to that obtained with optimum exposure by means of n'-bit 
shifting. The resultant image signal is supplied to the 
10 signal processor 7. Thereafter, the process proceeds to 
step S3 . 

In step S3, the signal processor 7 takes the first 
captured image as a reference image and sequentially selects 
one of image following the first image as a target image. 

15 The signal processor 7 detects a positional displacement 

(amount of motion) of the target image (one of second to N- 
th images) with respect to the reference image. Thereafter, 
the process proceeds to step S4 . 

In step S4, the signal processor 7 performs the correct 

20 image estimation process, based on the positional 

displacement of the N-th captured image relative to the 
reference image detected in step S3. Thereafter, the 
process proceeds to step S5 . In this correct image 
estimation process the details of which will be described 

25 later, the signal processor 7 produces a displacement- 
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compensated sharp image (output image) including all G, R, 
and B signals for each pixel and supplies the resultant 
image signal to one or both of the D/A converter. 9 and codec 
12 . 

5 In step S5, the output image is displayed on the 

monitor 11, and the output image is stored in the memory 13 
such as a flash memory. Thus, the process is completed. 
More specifically, in step S5, the image signal supplied in 
step S4 from the signal processor 7 to the D/A converter 9 

10 is converted into an analog signal and supplied to the video 
converter 10. Furthermore, in step S5, the video encoder 10 
converts the analog image signal supplied from the D/A 
converter 9 into a video signal in a form capable of being 
displayed on the monitor 11. The resultant video signal is 

15 supplied to the monitor 11. Furthermore, in step S5, the 
monitor 11 displays the image in accordance with the video 
signal supplied from the video encoder 10. Furthermore, in 
step S5, the image signal supplied in step S4 from the 
signal processor 7 to the codec 12 is encoded according to 

2 0 an encoding scheme such as that based on the JPEG standard 
or the MPEG standard, and the resultant image signal is 
stored in the memory 13 such as a flash memory. Thus, the 
process is exited. 

Fig. 3 is a diagram showing a manner in which pixels of 

2 5 the imaging device 4 are arranged. Although only some 
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pixels (6 pixels in a horizontal direction by 4 pixels in a 
vertical direction, and thus a total of 24 pixels) in an 
upper left area of the imaging device 4 are shown in Fig. 3, 
the remaining pixels are arranged in a similar manner. 
5 In Fig. 3, an X-Y coordinate system is defined such 

that the upper left corner of the imaging device 4 is 
employed as an origin, an X axis is taken in a horizontal 
direction (to the right), and a Y axis is taken in a 
vertical direction (upwardly) . The horizontal length and 

10 the vertical length of one pixel are set to be 1. The 

position (center position) of the pixel at an i-th position 
from the left end and at a j-th position from the upper end 
can be expressed as (i - 0.5, j - 0.5). In the imaging 
device 4 shown in Fig. 3, pixels are arranged in the form of 

15 a Bayer array. 

Pixels that provide G signals include a pixel GOO at a 
first position in the X direction and at a first position in 
the Y direction, a pixel G02 at a third position in the X 
direction and at the first position in the Y direction, a 

20 pixel G04 at a fifth position in the X direction and at the 
first position in the Y direction, a pixel Gil at a second 
position in the X direction and at a second position in the 
Y direction, and further a pixel G13, a pixel G15, a pixel 
G20, a pixel G22, a pixel G24, a pixel G31, a pixel G33, and 

25 a pixel G35 . 
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Pixels that provide R signals include a pixel R01 at 
the second position in the X direction and at the first 
position in the Y direction, a pixel R03 at a fourth 
position in the X direction and at the first position in the 
5 Y direction, a pixel ROB at a sixth position in the X 

direction and at the first position in the Y direction, a 
pixel R21 at the second position in the X direction and at a 
third first position in the Y direction, and further a pixel 
R23, and a pixel R25. 

10 Pixels that provide B signals include a pixel BIO at 

the first position in the X direction and at the second 
position in the Y direction, a pixel B12 at the third 
position in the X direction and at the second position in 
the Y direction, a pixel B14 at the fifth position in the X 

15 direction and at the second position in the Y direction, a 
pixel B30 at the first position in the X direction and at 
the fourth position in the Y direction, and further a pixel 
B32, and a pixel B34. 

For a sharp image with no blur due to a camera shake, a 

20 G signal, a R signal, and a B signal at a particular 

position (x, y) in the X-Y coordinate system of the imaging 
device 4 are respectively denoted by Lg(x, y) , Lr(x, y) , and 
Lb(x, y) . Furthermore, for a sharp image with no blur due 
to a camera shake, a G signal, a R signal, and a B signal of 

25 a (i-th, j-th) pixel located at an i-th position as counted 
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from the left end and at a j-th position as counted from the 
upper end are respectively denoted by Lg(i, j), Lr(i, j), 
and Lb(i, j). That is, Lg(x, y) , Lr(x, y) , and Lb(x, y) 
(Lg(i, j), Lr(i, j), and Lb(i, j)) respectively indicate the 
5 true green light intensity (data) , the true red light 

intensity (data) , the true blue light intensity (data) at 
the particular position (x, y) (at the (i-th, j-th) pixel) 
of an image having neither blur nor noise. Hereinafter, 
Lg(x, y) , Lr(x, y) , and Lb(x, y) (Lg(i, j), Lr(i, j), and 

10 Lb(i, j)) will be referred to as the true green light 

intensity Lg(x, y) (Lg(i, j)), the true red light intensity 
Lr(x, y) (Lr(i, j)), and the true blue light intensity Lb(x, 
y) (Lb(i, j)). When x = i - 0.5 and y = j - 0.5, Lg(x, y) = 
Lg(i, j), Lr(x, y) = Lr(i, j), and Lb(x, y) = Lb(i, j). 

15 Although in the present embodiment, the pixels of the 

imaging device 4 are arranged in the form of a Bayer array, 
the pixels may be arranged in another fashion. 

Variables ig, jg, ir, jr, ib, and jb associated with 
pixels of the G signal, the R signal, and the B signal are 

20 defined as follows. 

Variables ig and jg respectively indicate a position i 
in the X direction and a position j in the Y direction at 
which a pixel responsible for providing a G signal is 
located. That is, combinations of variables ig and jg are 

25 equal to combinations of variables i and j for pixels 
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responsible for providing the G signal. In the case in 
which pixel are arranged in the form of a Bayer array, 
variables ig and jg are given by variables i and j whose 
difference i - j is even. Note that, as a matter of course, 
5 the difference between variables ig and jg, that is, ig - jg, 
is also even. Thus, a "pixel located at an ig-th position 
and at a jg-th position" is a pixel that is responsible for 
providing the G signal. In a case in which pixels are 
arranged in the form of an array other than the Bayer array, 
10 variables ig and jg are given differently depending on the 
array . 

Variables ir and jr respectively indicate a position i 
in the X direction and a position j in the Y direction at 
which a pixel responsible for providing a R signal is 

15 located. That is, combinations of variables ir and jr are 
equal to combinations of variables i and j for pixels 
responsible for providing the R signal. In the case in 
which pixel are arranged in the form of the Bayer array, 
variables ir and jr are given by variables i and j that 

20 satisfy the constraint that the variable i is even and the 

difference between variables i and j, that is, i - j is odd. 
Note that, as a matter of course, the difference between 
variables ir and jr, that is, ir - jr, is also odd. Thus, a 
"pixel located at an ir-th position and at a jr-th position" 

25 is a pixel that is responsible for providing the R signal. 
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In a case in which pixels are arranged in the form of an 
array other than the Bayer array, variables ir and jr are 
given differently depending on the array. 

Variables ib and jb respectively indicate a position i 
5 in the X direction and a position j in the Y direction at 
which a pixel responsible for providing a B signal is 
located. That is, combinations of variables ib and jb are 
equal to combinations of variables i and j for pixels 
responsible for providing the B signal. In the case in 

10 which pixel are arranged in the form of the Bayer array, 
variables ib and jb are given by variables i and j that 
satisfy the constraint that the variable i is odd and the 
difference between variables i and j, that is, i - j is odd. 
Note that, as a matter of course, the difference between 

15 variables ib and jb, that is, ib - jb, is also odd. Thus, a 
"pixel located at an ib-th position and at a jb-th position" 
is a pixel that is responsible for providing the B signal. 
In a case in which pixels are arranged in the form of an 
array other than the Bayer array, variables ib and jb are 

20 given differently depending on the array. 

Values (pixel values) of color signals (a G signal, a R 
signal, and a B signal) output from respective pixels of the 
imaging device 4 shown in Fig. 3 are defined as follows. 

As described earlier, N images are taken by the imaging 

25 device 4, and thus N pixel values are obtained for each 
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pixel of the imaging device 4 . A pixel value of a pixel at 
an ig-th position and at a jg-th position of a k-th image (k 
= 1 to N) is denoted by Gobs(k, ig, jg) , a pixel value of a 
pixel at an ir-th position and at a jr-th position of a k-th 
5 image (k = 1 to N) is denoted by Robs(k, ir, jr) , and a 

pixel value of a pixel at an ib-th position and at a jb-th 
position of a k-th image (k = 1 to N) is denoted by Bobs (k, 
ib, jb) . For example, a pixel value of a pixel GOO of a 
first image is given by Gobs (1, 1, 1), and a pixel value of 

10 a pixel G04 of a second image is given by Gobs (2, 5, 1) . 

Herein, and also in the following description, k denotes an 
integer in the range from 1 to N, unless otherwise specified. 

In other words, pixel values Gobs(k, ig, jg) , Robs(k, 
ir, jr) , Bobs(k, ib, jb) respectively indicate pixel values 

15 obtained by pixels G(jg - 1) (ig - 1), R(jr - 1) (ir - 1), and 
B ( jb - 1) (ib - 1) . 

Fig. 4 shows an example of detailed configuration of a 
part of the signal processor 7 shown in Fig. 1. The signal 
processor 7 includes a frame memory 22, a motion detector 23, 

20 a calculation circuit 24, and a controller 25. The frame 
memory 22 includes frame memories 22-1 to 22 -N, and the 
motion detector 23 includes motion detectors 23-1 to 23- (N - 
1) . 

As described earlier, the A/D converter 6 supplies N 
25 captured images to the frame memory 22. The frame memory 
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22-1 stores a first captured image supplied from the A/D 
converter 6. The frame memory 22-2 stores a second captured 
image supplied from the A/D converter 6. Similarly, the 
frame memory 22 -k stores a k-th captured image supplied from 
the A/D converter 6 (where k = 3 to N) . 

The frame memory 22-1 supplies the first captured image 
stored therein to the calculation circuit 24 and the motion 
detectors 23-1 to 23- (N - 1) with proper timing. The frame 
memory 22-2 supplies the second captured image stored 
therein to the calculation circuit 24 and the motion 
detector 23-1 with proper timing. Similarly, the frame 
memory 22 -k supplies the k-th captured image stored therein 
to the calculation circuit 24 and the motion detector 23- (k 
- 1) with proper timing. 

The motion detector 23 detects a positional 
relationship between two captured images. More specifically, 
the motion detector 23 takes the first captured image as a 
reference image and one of captured images following the 
first captured image as a target image, and the motion 
detector 23 detects a positional displacement (amount of 
motion) of the target image (one of second to N-th images) 
with respect to the reference image. For example, a camera 
shake can cause a positional displacement . 

The first captured image used as the reference image is 
supplied to the motion detector 23-1 from the frame memory 
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22-1, and the second captured image used as the target image 
is supplied to the motion detector 23-1 from the frame 
memory 22-2. 

The motion detector 23-1 detects the correspondence 
5 between the position of each pixel (or each block when the 
image is divided into a plurality of blocks) of the second 
captured image and the position of the first captured image 
by determining conversion parameters (9, T2x, T2y, S2) of 
equation (1) representing the positional relationship 
10 between the first captured image and the second captured 

image, where 02 denotes a rotation angle, S2 denotes a scale, 
and (T2x, T2y) denotes a parallel displacement. The 
resultant conversion parameters are supplied to the 
calculation circuit 24. 
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Equation (1) is in the form of a so-called affine 
transformation, in which (X2, Y2) represents the position of 
a pixel of the second captured image, and (XI (2 > , Yl (2 )) 
represents the corresponding position on the first captured 
2 0 image. Note that a subscript (2) is used to indicate a 
converted position of a position on the second captured 
image . 

The conversion parameters (02, T2x, T2y, S2) represent 
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that a position (X2, Y2) on the second captured image is 
displaced with respect to a corresponding position (XI, Yl) 
on the first captured image by an angle 02 due to a camera 
shake, enlarged (or reduced if S2 < 1) by a factor of S2 due 
5 to movement of the digital camera 1 toward a subject, and 
displaced parallel to the subject by (T2x, T2y) . In most 
cases, of various displacement components of camera shake, a 
component in a direction parallel to the plane of the 
imaging device 4 has a significant effect, but a 

10 displacement component in a direction from the digital 

camera 1 to the subject is small enough to neglect it. That 
is, S2 can be approximated as S2 = 1 . 

The first captured image used as the reference image is 
supplied to the motion detector 23-2 from the frame memory 

15 22-1, and the third captured image used as the target image 
is supplied to the motion detector 23-2 from the frame 
memory 22 -3 . 

The motion detector 23-2 detects the correspondence 
between the position of each pixel (or each block when the 

20 image is divided into a plurality of blocks) of the third 

captured image and the position of the first captured image 
by determining conversion parameters (93, T3x, T3y, S3) of 
equation (1) representing the positional relationship 
between the first captured image and the second captured 

25 image, where 03 denotes a rotation angle, S3 denotes a scale, 
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and (T3x, T3y) denotes a parallel displacement. The 



resultant conversion parameters are supplied to the 



calculation circuit 24. 




= S3 



cos(03) -sin(03) X3 T3x 



sin(03) cos((93) Y3 T3y 
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Equation (2) is also in the form of an affine 
transformation, in which (X3, Y3) represents the position of 
a pixel of the second captured image, and (Xl (3)/ Yl (3) ) 
represents the corresponding position on the first captured 
image. Note that a subscript (3) is used to indicate that a 
position of the third captured image is converted to a 
position with the subscript (3). 

The conversion parameters (03, T3x, T3y, S3) represent 
that a position (X3, Y3) on the third captured image is 
displaced with respect to a corresponding position (XI, Yl) 
on the first captured image by an angle 03 due to a camera 
shake, enlarged (or reduced if S3 < 1) by a factor of S3 due 
to movement of the digital camera 1 toward the subject, and 
displaced parallel to the subject by (T3x, T3y) . In most 
cases, of various displacement components of camera shake, a 
component in a direction parallel to the plane of the 
imaging device 4 has a significant effect, but a 
displacement component in a direction from the digital 
camera 1 to the subject is small enough to neglect it. That 
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10 



15 



is, S2 can be approximated by S3 = 1 . 

Similarly, the first captured image used as the 
reference image is supplied to the motion detector 23- (k - 
1) from the frame memory 22-1, and the k-th captured image 
used as the target image is supplied to the motion detector 
23- (k - 1) from the frame memory 22-k. 

The motion detector 23- (k - 1) detects the 
correspondence between the position of each pixel (or each 
block when the image is divided into a plurality of blocks) 
of the k-th captured image and the position of the first 
captured image by determining conversion parameters (6k, Tkx, 
Tky, Sk) of equation (1) representing the positional 
relationship between the first captured image and the second 
captured image, where 0k denotes a rotation angle, Sk 
denotes a scale, and (Tkx, Tky) denotes a parallel 
displacement. The resultant conversion parameters (0k, Tkx, 
Tky, Sk) are supplied to the calculation circuit 24. 
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20 Equation (3) is also in the form of an affine 

transformation, in which (Xk, Yk) represents the position of 
a pixel of the second captured image, and (XI (k) , Yl< k) ) 
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represents the corresponding position on the first captured 
image. Note that a subscript (k) is used to indicate that a 
position of the second captured image is converted to a 
position with the subscript (k) . 
5 The conversion parameters (9k, Tkx, Tky, Sk) represent 

that a position (Xk, Yk) on the k-th captured image is 
displaced with respect to a corresponding position (XI, Yl) 
on the first captured image by an angle 0k due to a camera 
shake, enlarged (or reduced if Sk < 1) by a factor of Sk due 

10 to movement of the digital camera 1 toward the subject, and 
displaced parallel to the subject by (Tkx, Tky) . In most 
cases, of various displacement components of camera shake, a 
component in a direction parallel to the plane of the 
imaging device 4 has a significant effect , but a 

15 displacement component in a direction from the digital 

camera 1 to the subject is small enough to neglect it. That 
is, Sk can be approximated by Sk = 1. 

Instead of determining the conversion parameters (0k, 
Tkx, Tky, Sk) from the positional relationship of the k-th 

20 captured image with respect to the first captured image in 
the above-described manner, the conversion parameters (0k, 
Tkx, Tky, Sk) may be mechanically determined from a signal 
output from an acceleration sensor disposed on the digital 
camera 1 . 

25 N captured images are supplied to the calculation 
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circuit 24 from the frame memories 22-1 to 22 -N, and the 
conversion parameters (0k, Tkx, Tky, Sk) representing the 
positional relationship between the first captured image and 
the k-th captured image are supplied to the calculation 
5 circuit 24 from the motion detectors 23-1 to 23- (N - 1). 

The calculation circuit 24 produces a displacement- 
compensated sharp image signal (including a G signal 
component, R signal component, and B signal component) by 
performing a correct image estimation process, which will be 

10 described later, based on the positional relationships of 

the second to N-th captured images supplied from the motion 
detectors 23-1 to 23- (N - 1) with respect to the first 
captured image, and the calculation circuit 24 supplies 
resultant displacement-compensated image signal to the D/A 

15 converter 9 or the codec 12 . In each of the N captured 
images supplied from the A/D converter 6 to the signal 
processor 7, each pixel has one of G, R, and B signals. On 
the other hand, in the compensated image signal produced by 
the calculation circuit 24, each pixel has all of G, R, and 

20 B signals (data) . 

Under the control of the CPU 15, the controller 25 
controls various parts of the signal processor 7, such as 
the frame memories 22-1 to 22-N, the motion detectors 23-1 
to 23- (N - 1), the calculation circuit 24, etc. Instead of 

25 the controller 25, the CPU 15 (Fig. 1) may control parts of 
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the signal processor 7, such as the frame memories 22-1 to 
22-N, the motion detectors 23-1 to 23- (N - 1), and 
calculation circuit 24. In this case, the controller 25 is 
not necessary. 

5 In the single-plane sensor using the Bayer array, the 

number of pixels for R or B signals is less than the number 
of pixels for the G signal. This can cause the R signal or 
the B signal in the image output from the signal processor 7 
to have a greater error than an error the G signal has. In 

10 such a case, a lowpass filter, which limits only a high- 
frequency component of the color signal without limiting the 
luminance signal, is disposed at a stage following the 
calculation circuit 24 thereby removing noise. 

If the calculation circuit 24 is designed to be capable 

15 of processing images at a high speed on a real-time basis 
while the images are sequentially captured, the processing 
can be performed using a less number of frame memories 22-1 
to 22-N and a less number of motion detectors 23-1 to 23- (N 
- 1) , and thus the signal processor 7 can be constructed in 

2 0 a simpler form with less complexity. 

Now, a correct image estimation process performed by 
the signal processor 7 according to a first embodiment of 
the present invention is described. 

Although not shown in the figures, an on-chip lens is 

25 disposed directly on each pixel (photosensing part of each 
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pixel) of the imaging device 4. Each on-chip lens focuses 
all light originating from a subject and incident on the on- 
chip lens into one point. Therefore, each pixel located 
immediately under its own on-chip lens receives the integral 
5 of all light incident on the on-chip lens. This results in 
an improvement in sensitivity of each pixel. 

Therefore, data (light intensity) received by each 
pixel of the imaging device 4 is not the light intensity of 
light originating from a subject and incident on a point 

10 (that is, not point-sampled data) but the integral of light 
originating from the subject and incident on a finite area 
of a plane (photosensing plane) . 

In the first embodiment, the characteristic of the on- 
chip lens is precisely formulated so that a sharp image 

15 (output image) is produced via a correct image estimation 
process. Note that in the conventional techniques, it is 
assumed that data is point-sampled at each pixel. However, 
in an actual imaging device 4, as described above, each 
pixel receives the integral of light incident on a finite 

20 area of a plane, and thus the assumption of point -sampling 

results in an error in estimation (restoration) of a correct 
image . 

Now, for the first image stored in the frame memory 22- 
1, the relationship between the pixel value obtained at each 
25 pixel and light incident via the on-chip lens is formulated. 
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For example, for a pixel GOO at a (1st, 1st) position 
in upper left corner shown in Fig. 3, the relationship 
between the pixel value Gobs(l, 1, 1) of that pixel and the 
true green light intensity Lg(x, y) is given by a following 
5 equation (4) . 



The pixel GOO in upper left corner of the photosensor 4 
serving as the single-plane sensor is covered with a green 
filter such that only a green component of incident light 

10 can reach the pixel GOO. Because of the provision of the 
on-chip lens, light incident on a rectangular region whose 
corners are at coordinates (0, 0), (0, 1), (1, 0), and (1, 
1) in Fig. 3 reaches the pixel GOO. That is, all of light 
incident on the rectangular region whose corners are at 

15 coordinates (0, 0), (0, 1), (1, 0), and (1, 1) is sensed by 
the pixel GOO located at the (1st, 1st) position. 

The left-hand part in equation (4) represents the 
integral of the true green light intensity Lg(x, y) 
(incident) at a position (x, y) in the coordinate system of 

20 the first captured image taken over a rectangular range 

whose corners are located at (0, 0), (0, 1), (1, 0), and (1, 
1) , that is, over a range defined by 0 < x < 1 and 0 < y < 1. 
The right-hand part of equation (4) represents the pixel 
value Gobs(l, 1, 1) obtained (observed) by the pixel at the 




1 



1 



Lg(x, y)dxdy = Gobsd, 1, 1) 



■ • • (4) 
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(1st, 1st) position of the first image. Thus, equation (4) 
represents that the integral of the true green light 
intensity Lg(x, y) incident on the light sensing plane of 
the pixel GOO taken over the light sensing plane of the 
5 pixel GOO is equal to the pixel value Gobsd, 1, 1) observed 
by the pixel GOO for the first image. 

Depending on the characteristic of the on-chip lens, 
light originating from a subject and being incident on a 
peripheral area (close to edges) of each pixel is not 

10 focused. In such a case, the range over which the integral 
of the left-hand part of equation (4) is calculated is 
slightly reduced. For example, the integral is calculated 
over a range whose corners are at (0.1, 0.1), (0.1, 0.9), 
(0.9, 0.1), and (0.9, 0.9). 

15 In many digital cameras, to avoid false color, which is 

a drawback of a single-plane sensor, an optical lowpass 
filter is disposed on each pixel. In this configuration, 
v each pixel receives light incident on an area slightly 
greater than a rectangular area of the pixel . In such a 

2 0 case, the rectangular range over which the integral of the 
left-hand part of equation (4) is calculated is slightly 
expanded . 

The discussion will be continued by taking as an 
example a (2nd, 1st) pixel R01 located immediately adjacent 
25 to the right of the pixel GOO in Fig. 3. 
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For the pixel R01 at the (2nd, 1st) position, the 
relationship between the pixel value Robsd, 2, 1) of that 
pixel and the true green light intensity Lg(x, y) is given 
by a following equation (5) . 

fj f 2 Lr(x,y)dxdy = Robs (1,2, 1) 

5 J 0 J ^ ■ • • (5) 

The pixel R01 in upper left corner of the photosensor 4 
serving as the single-plane sensor is covered with a red 
filter such that only a red component of incident light can 
reach the pixel R01. Because of the provision of the on- 

10 chip lens, light incident on a rectangular region whose 

corners are at coordinates (1, 0), (1, 1), (2, 0), and (2, 
1) in Fig. 3 reaches the pixel R01. That is, all of light 
incident on the rectangular region whose corners are at 
coordinates (1, 0), (1, 1), (2, 0), and (2, 1) is sensed by 

15 the pixel R01 located at the (2nd, 1st) position. 

The left-hand part in equation (5) represents the 
integral of the true red light intensity Lr (x, y) (incident) 
at a position (x, y) in the coordinate system of the first 
captured image over a rectangular range whose corners are 

20 located at (1, 0), (1, 1), (2, 0), and (2, 1), that is, over 
a range defined by 1 < x < 2 and 0 < y < 1. The right-hand 
part of equation (5) represents the pixel value Robs(l, 2, 
1) obtained (observed) by the pixel at the (2nd, 1st) 
position of the first image. Thus, equation (5) represents 
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that the integral of the true red light intensity Lr(x, y) 
incident on the light sensing plane of the pixel R01 over 
the light sensing plane of the pixel R01 is equal to the 
pixel value Robs ( 1 , 2, 1) observed by the pixel R01 for the 
5 first image. 

Depending on the characteristic of the on-chip lens, 
light originating from a subject and being incident on an a 
peripheral area (close to edges) of each pixel is not 
focused. In such a case, the range over which the integral 

10 of the left-hand part of equation (5) is calculated is 

slightly reduced. For example, the integral is calculated 
over a range whose corners are at (1.1, 0.1), (1.1, 0.9), 
(1.9, 0.1), and (1.9, 0.9). 

In many digital cameras, to avoid false color, which is 

15 a drawback of a single-plane sensor, an optical lowpass 

filter is disposed on each pixel. In this configuration, 
each pixel receives light incident on an area slightly 
greater than a rectangular area of the pixel. In such a 
case, the rectangular range over which the integral of the 

20 left-hand part of equation (5) is calculated is slightly 
expanded . 

Similar equations are defined for the other pixels of 
the first captured image. 

That is, for a pixel G(jg - 1) (ig - l)at a (ig-th, jg- 
25 th) position shown in Fig. 3, the relationship between the 
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pixel value Gobs(l, ig, jg) of that pixel and the true green 
light intensity Lg(x, y) is given by a following equation 

(6) . 

J* jg pig 
. .J. 1 Lg(x, y)dxdy = GobsO, ig, jg) 
Jg-I lg-1 • • • (6) 

5 Equation (6) represents that the pixel value Gobs (1, ig, 

jg) is equal to the integral of the true green light 
intensity Lg(x, y) over a rectangular range whose corners 
are located at (ig - 1, jg - 1) , (ig - 1, jg) , (ig, jg - 1) , 
and (ig, jg) in the coordinate system shown in Fig. 3, that 
10 is, over a range defined by ig - 1 < x < ig and jg - 1 < y < 

jg- 

For a pixel R(jr - 1) (ir - l)at a (ir-th, jr-th) 
position shown in Fig. 3, the relationship between the pixel 
value Robsd, ir, jr) of that pixel and the true red light 
15 intensity Lr(x, y) is given by a following equation (7). 

//-X-i Lr(x.y)dxdy = Robsd. ir, jr) (?) 

Equation (7) represents that the pixel value Robs(l, ir, 
jr) is equal to the integral of the true red light intensity 
Lr(x, y) over a rectangular range whose corners are located 
20 at (ir - 1, jr - 1) , (ir - 1, jr) , (ir, jr - 1) , and (ir, 

jr) in the coordinate system shown in Fig. 3, that is, over 
a range defined by ir - 1 < x < ir and jr- 1 < y < jr. 
For a pixel B(jb - 1) (ib - l)at a (ib-th, jb-th) 
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position shown in Fig. 3, the relationship between the pixel 
value Bobs (1, ib, jb) of that pixel and the true blue light 
intensity Lib(x, y) is given by a following equation (8) . 



jb) is equal to the integral of the true blue light 
intensity Lb(x, y) over a rectangular range whose corners 
are located at (ib - 1, jb - 1) , (ib - 1, jb) , (ib, jb - 1) , 
and (ib, jb) in the coordinate system shown in Fig. 3, that 
10 is, over a range defined by ib - 1 < x < ib and jb - 1 < y < 



In practice, as described earlier, the image signal 
(the observed pixel value) includes noise E, which is gained 
up by a factor of Mk . If noise components E x Mk are taken 
15 into account, formulas (6), (7), and (8) are respectively 
rewritten into formulas (9) , (10) , and (11) indicating 
constraints that light intensity integrals should satisfy. 




• • ■ (8) 



5 



Equation (8) represents that the pixel value Bobs(l, ib, 



jb. 



Xg-1 Xg-1 Lg(x, y)dxdy-Gobs(1, ig, jg) < M1 xE 



(9) 




Lr (x, y)dxdy-Robs(1, ir, jr) < M1 xE 



• • • (10) 



Jjb-1 Jib-1 Lb(x,y)dxdy-Bobs(1, ib, jb) < M1 xE 



r jb r ib 



20 



(11) 



where |x| denotes the absolute value of x. 
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Formula (9) represents that the difference between the 
integral of light intensity Lg(x, y) and the observed pixel 
value Gobsd, ig, jg) is equal to or less than a maximum 
possible value Ml x E of noise. Inequalities (10) and (11) 
5 represent similar constraints. 

In the following description, the k-th captured image 
will also be referred to simply as the k-th image. 

Now, for the second image stored in the frame memory 
22-2, as for the first image, the relationship between the 
10 pixel value obtained (observed) at each pixel and light 

incident via the on-chip lens is represented in the form of 
a mathematical expression. 

The coordinate system of the second and further 
following captured images is converted to the coordinate 
15 system of the first captured image based on the conversion 
parameters (Gk, Tkx, Tky, Sk) (k = 2 to N) detected by the 
motion detector 23 . 

For a pixel GOO at a (1st, 1st) position of the second 
image, the relationship between the pixel value Gobs (2, 1, 
20 1) of that pixel and the true green light intensity Lg(x, y) 
is given by a following equation (12) . 



The pixel GOO in upper left corner of the photosensor 4 
is covered 'with a green filter such that only a green 




• • ■ (12) 
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component of incident light can reach the pixel GOO. 
Because of the provision of the on-chip lens, light incident 
on a rectangular region whose corners are at coordinates (0, 

0) , (0, 1), (1, 0), and (1, 1) in the coordinate system of 
5 the second image reaches the pixel GOO. 

If positions (0, 0), (0, 1), (1, 0), and (1, 1) in the 
coordinate system of the second image are converted to 
positions in the coordinate system of the first image based 
on the conversion parameters (02, T2x, T2y, S2) , the 
10 resultant positions are given as (0< 2 ), 0 (2 )), (0 (2 ), 1(2))/ 

(1(2)/ 0(2))/ and (1(2)/ 1(2))- That is, all of light incident 
on the rectangular region whose corners are at coordinates 
(0 ( 2), 0(2)), (0(2), 1(2))/ (1(2)/ 0(2)) and (1 (2 )/ 1(2)) is sensed 
by the pixel GOO located at the (1st, 1st) position. 
15 Hereinafter, when a position (x, y) of the second image is 
converted into a position in the coordinate system of the 
first image, a resultant position will be referred to as P(2, 
x, y) . 

The left-hand part in equation (12) represents the 
20 integral of the true green light intensity L»g(x, y) over a 
rectangular region whose corners are at P(2, 0, 0), P(2, 0, 

1) , P(2, 1, 0), and P(2, 1, 1), that is, over a rectangular 
range whose corners are located at coordinates (0 (2 ), 0 (2 )), 
(0 (2 ), 1( 2 )), (1( 2 ), 0( 2) ), and (1( 2 )/ 1(2)) in the coordinate 

25 system of the first image. The right-hand part of equation 
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(12) represents the pixel value Gobs (2, 1, 1) obtained at 
the pixel at the (1st, 1st) position of the second image. 
Equation (12) represents that the pixel value Gobs (2, 1, 1) 
observed at the pixel GOO of the second image is equal to 
5 the integral of the true green light intensity L.g(x, y) 

incident on a rectangular region whose corners are located 
at (0(2), 0(2)), (0(2), 1<2)), (1(2)/ 0(2)), and (1 (2 ), 1(2)) in the 
coordinate system of the first image. In equation (12), 
Ifdxdy represents the integral taken over a rectangular 

10 region whose corners are at (0 (2 ), 0 (2 )), (0 (2 ), 1(2))/ (1(2), 
0{ 2 )), and (1(2)/ 1(2))- In equation (13) and other similar 
equations appearing in the following description, Ifdxdy will 
be also used to represent a similar integral. 

Depending on the characteristic of the on-chip lens, 

15 light originating from a subject and being incident on an a 
peripheral area (close to edges) of each pixel is not 
focused. In such a case, the range over which the integral 
in the left-hand part of equation (12) is taken is slightly 
shrunk as in the first image. 

20 In many digital cameras, to avoid false color, which is 

a drawback of a single-plane sensor, an optical lowpass 
filter is disposed on each pixel. In this configuration, 
each pixel receives light incident on an area slightly 
greater than a rectangular area of the pixel. In such a 

25 case, the rectangular range over which the integral of the 
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left-hand part of equation (12) is calculated is slightly 
expanded . 

The discussion will be continued by taking as an 
example the (2nd, 1st) pixel R01 located immediately 
5 adjacent to the right of the pixel GOO. 

For the pixel R01 at the (2nd, 1st) position of the 
second image, the relationship between the pixel value 
Robs (2, 2, 1) of that pixel and the true red light intensity 
Lr(x, y) is given by a following equation (13). 



The pixel R01 in upper left corner of the photosensor 4 
is covered with a red filter such that only a red component 
of incident light can reach the pixel R01 . Because of the 
provision of the on-chip lens, light incident on a 

15 rectangular region whose corners are at coordinates (1, 0) , 
(1, 1), (2, 0), and (2, 1) in the coordinate system of the 
second image reaches the pixel R01. 

If positions (1, 0), (1, 1), (2, 0), and (2, 1) in the 
coordinate system of the second image are converted to 

20 positions in the coordinate system of the first image based 
on the conversion parameters (02, T2x, T2y, S2), the 
resultant positions are given as (1(2)/ 0(2))/ (1(2)/ 1(2))/ 
(2 (2)/ 0(2))/ and (2< 2 )/ 1(2))- That is, all of light incident 
on the rectangular region whose corners are at coordinates 



10 




• • • (13) 



- 52 - 

S04P0831 

(1(2)/ 0(2)), (1(2)/ 1(2))/ (2(2), 0(2)), and (2 (2) , 1< 2 )) is sensed 
by the pixel R01 located at the (2nd, 1st) position. 

The left-hand part in equation (13) represents the 
integral of the true red light intensity Lr(x, y) over a 
5 rectangular region whose corners are at P(2, 1, 0), P(2, 1, 
1), P(2, 2, 0), and P(2, 2, 1), that is, over a rectangular 
range whose corners are located at coordinates (1( 2 ), 0 (2 )), 
(1(2), 1(2)), (2(2), 0(2)), and (2 (2 ), 1< 2 >) in the coordinate 
system of the first image. The right-hand part of equation 

10 (13) represents the pixel value Robs (2, 2, 1) obtained at 
the pixel at the (2nd, 1st) position of the second image. 
Equation (13) represents that the pixel value Robs (2, 2, 1) 
observed at the pixel R01 of the second image is equal to 
the integral of the true green light intensity Lr(x, y) 

15 incident on a rectangular region whose corners are located 

at (1(2), 0 (2) ), (1(2), 1(2))/ (2(2)/ 0(2)), and (2 (2) , 1< 2 )) in the 
coordinate system of the first image. 

Depending on the characteristic of the on-chip lens, 
light originating from a subject and being incident on an a 

20 peripheral area (close to edges) of each pixel is not 

focused. In such a case, the range over which the integral 
in the left-hand part of equation (13) is taken is slightly. 

In many digital cameras, to avoid false color, which is 
a drawback of a single-plane sensor, an optical lowpass 

25 filter is disposed on each pixel. In this configuration, 
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each pixel receives light incident on an area slightly 
greater than a rectangular area of the pixel . In such a 
case, the rectangular range over which the integral of the 
left-hand part of equation (13) is calculated is slightly 
5 expanded . 

Similar equations are defined for the other pixels of 
the second captured image . 

That is, for a pixel G(jg - 1) (ig - l)at a (ig-th, jg- 
th) position of the second image, the relationship between 
10 the pixel value Gobs (2, ig, jg) of that pixel and the true 
green light intensity Lg(x, y) is given by a following 
equation (14) . 
p jg(2) i S (2) 

Ji*-i Jia-i Lg(x, y)dxdy = Gobs(2, ig, jg) 

Jjg l (2) ^ 1 g "(2) ... (14) 

Equation (14) represents that the pixel value Gobs (2, 
15 ig, jg) is equal to the integral of the true green light 
intensity Lg(x, y) over a rectangular range whose corners 
are located at (ig - 1, jg - 1) , (ig - 1, jg) , (ig, jg - 1) , 
and (ig, jg) in the coordinate system of the second image, 
that is, over a rectangular range whose corners are located 

20 at (ig - i (2) , jg - i< 2 >), (ig - i(2), jg< 2 )), (ig<2>, jg - i< 2 >), 

and (ig<2), jg<2>) in the coordinate system of the first image. 

That is, for a pixel R(jr - 1) (ir - l)at a (ir-th, jr- 
th) position of the second image, the relationship between 
the pixel value Robs (2, ir, jr) of that pixel and the true 
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red light intensity Lr(x, y) is given by a following 
equation (15) . 

J.JIC C ' r (2) 
. 1 J. 1 Lr (x, y)dxdy = Robs(2, ir, jr) 
Jg- 1 ( 2 ) ^ir-la) ... (15) 

Equation (15) represents that the pixel value Robs (2, 
ir, jr) is equal to the integral of the true red light 
intensity Lr (x, y) over a rectangular range whose corners 
are located at (ir - 1, jr - 1) , (ir - 1, jr) , (ir, jr - 1) , 
and (ir, jr) in the coordinate system of the second image, 
that is, over a rectangular range whose corners are located 
at (ir - 1 {2)/ jr - 1 (2 )), (ir - 1 (2) , jr (2) ), (ir (2)/ jr - 1(2)), 
and (ir {2 ), jr (2) ) in the coordinate system of the first image. 

That is, for a pixel B(jb - 1) (ib - l)at a (ib-th, jb- 
th) position of the second image, the relationship between 
the pixel value Bobs (2, ib, jb) of that pixel and the true 
red light intensity Lb(x, y) is given by a following 
equation (16) . 

ih 1 r. 1 Lb(x,y)dxdy = Bobs(2, ib, jb) 

JD-l(2)«Mb-l(2) ... (16) 

Equation (16) represents that the pixel value Bobs (2, 
ib, jb) is equal to the integral of the true blue light 
intensity Lb(x, y) over a rectangular range whose corners 
are located at (ib - 1,-jb - 1), (ib - 1, jb) , (ib, jb - 1) , 
and (ib, jb) in the coordinate system of the second image, 
that is, over a rectangular range whose corners are located 
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at (ib - 1(2), jb - 1(2))/ (ib - 1 (2)/ jb (2) ), (ib (2)/ jb - 1(2)), 
and (ib( 2 ), jb< 2 )) in the coordinate system of the first image 

In practice, as described earlier, the image signal 
(the observed pixel value) includes noise E, which is gained 
up by a factor of Mk . If noise components E x Mk are taken 
into account, formulas (14) , (15) , and (16) are respectively 
rewritten into formulas (17) , (18) , and (19) indicating 
constraints that light intensity integrals should satisfy. 

J.Jg( 2 ) /* 1 £ (2) 
\„ 1 J\„ 1 Lg(x, y)dxdy-Gobs(2, ig, jg) 
Jg— I tn\ ^ Ig— I <n\ 



J.jr (2) p i r (2) 
, I. , Lr (x, y)dxdy-Robs (2, i r, jr) 
xu , jr-1 (2) Jir-1 (2) 



/if? Jlh b ? Lb(x,y)dxdy-Bobs(2, ib, jb) 

^ Jb-l (2) *^ ID I (2 ) 



< M2xE 

< M2xE 

< M2xE 



(17) 



(18) 



(19) 



where |x| denotes the absolute value of x. 

Formula (17) represents that the difference between the 
integral of light intensity Lg(x, y) and the observed pixel 
15 value Gobs (2, ig, jg) is equal to or less than a maximum 

possible value M2 x E of noise. Inequalities (18) and (19) 
represent similar constrains. 

Now, referring to Fig. 5, the relationship between a 
position (x, y) of the second image and a position P(2, x, 
20 y) obtained by converting the position (x, y) into 
coordinate system of the first image. 

A pixel 31 in a rectangular region whose four corners 
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are at (i - 1, j - 1) , (I - 1, j ) , (i, 1 - 1), and (i, j) in 
the coordinate system of the second image is shown on the 
left-hand side of Fig. 5. 

The pixel 31 on the left-hand side is converted to a 
5 pixel 31' in the coordinate system of the first image, as 
shown on the right-hand side of Fig. 5. Note that light of 
the same part of a subject (for example, a landscape scene) 
falls on the pixel 31 on the left-hand side of Fig. 5 and 
the pixel 31 1 on the right-hand side of Fig. 5. 

10 The pixel value of each pixel of second and following 

captured images is determined as the integral of light in a 
particular region in the first image employed as the 
reference image. By properly setting the integration region, 
the first to N-th captured images having displacement are 

15 aligned into a correct position (with reference to the first 
captured image) . 

A position (i - 1, j - 1) in the coordinate system of 
the second image is converted to a position P(2, i - 1, j - 
1) in the coordinate system of the first image as a result 

20 of the conversion based on the conversion parameters (02, 
T2x, T2y, S2) . A position (i - 1, j) in the coordinate 
system of the second image is converted to a position P(2, i 
- 1, j) in the coordinate system of the first image as a 
result of the conversion based on the conversion parameters 

25 (02, T2x, T2y, S2). Similarly, positions (i, j - 1) and (i, 
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j ) in the coordinate system of the second image are 
converted to positions P(2, i, j - 1) and P(2, i, j) in the 
coordinate system of the first image as a result of the 
conversion based on the conversion parameters (02, T2x, T2y, 
5 S2) . Hereinafter, the X coordinate axis and the Y 

coordinate axis in the coordinate system of the first image 
will be simply referred to as X and Y, respectively. 

As with the second image, positions (i - 1, j - 1), (i 

- 1, j), (i, j - 1) and (i, j) in the coordinate system of 
10 the k-th (k = 3 to N) image are converted to position P(k, i 

- 1, j - 1), P(k, i - 1, j), P(k, i, j - 1) and P(k, i, j) 
in the coordinate system of the first image as a result of 
the conversion based on the conversion parameters (0k, Tkx, 
Tky, Sk) . 

15 In the k-th image (k = 3 to N) , if noise E with a 

magnitude gained up by a factor Mk is taken into account, 
formulas (20) and (22) similar to formulas (17) and (19) 
indicating constraint imposed on the integral of light 
density are obtained. 
Jg(k) T ig (k) 



J iff _i U(x, y)dxdy-Gobs(k, ig, jg) 

:u , Jg l(k)*"g "00 



< Mk xE 

• • ■ (20) 



Formula (20) represents that the pixel value Gobs(k, ig, 
jg) obtained by a pixel located at a (ig-th, jg-th) position 
of the k-th image is equal to the integral of the true green 
light intensity Lg(x, y) taken over a rectangular range 
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whose corners are located at P(k, ig - 1, jg - 1) , P(k, ig - 
1, jg) / P(k, ig, jg - 1), and P(k, ig, jg) , that is, over a 
rectangular range whose corners are located at (ig - l<k)/ 39 
- l(k))/ (ig - l{k)/ jg(k))/ (ig<k)/ jg - l(k>), and (ig ( k), jg<k)) 
5 in the coordinate system of the first image, with a maximum 
possible error equal to noise E with a magnitude gained up 
by a factor Mk. That is, the (absolute value of) difference 
between the integral of true green light intensity Lg(x, y) 
and the observed pixel value Gobs(k, ig, jg) is equal to or 
10 less than a maximum possible value of noise. Note that in 
formula (20), |x| denotes the absolute value of x. 



Formula (21) represents that the pixel value Robs(k, ir, 
jr) obtained by a pixel located at a (ir-th, jr-th) position 

15 of the k-th image is equal to the integral of the true red 
light intensity Lr(x, y) taken over a rectangular range 
whose corners are located at P(k, ir - 1, jr - 1), P(k, ir - 
1, jr) , P(k, ir, jr - 1), and P(k, ir, jr) , that is, over a 
rectangular range whose corners are located at (ir - l(k)/ jr 

20 - l ( k)), (ir - l (k ), jr (k j), (ir (k ), jr - l(k>), and (ir (k j , jr (k) ) 
in the coordinate system of the first image, with a maximum 
possible error equal to noise E with a magnitude gained up 
by a factor Mk. That is, the (absolute value of) difference 
between the integral of true red light intensity Lr(x, y) 




Lr (x, y)dxdy-Robs(k, ir, jr) < MkxE 



• ■ ■ (21) 
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and the observed pixel value Robs(k, ir, jr) is equal to or 
less than a maximum possible value of noise. Note that in 
formula (21), |x| denotes the absolute value of x. 



jb) obtained by a pixel located at a (ib-th, jb-th) position 
of the k-th image is equal to the integral of the true blue 
light intensity Lb(x, y) taken over a rectangular range 
whose corners are located at P(k, ib - 1, jb - 1) , P(k, ib - 

10 1, jb) , P(k, ib, jb - 1) , and P(k, ib, jb) , that is, over a 
rectangular range whose corners are located at (ib - l< k ), jb 
- l (k ))# (ib - l (k ), jb( k ))/ (ib (k) , jb - l< k) ), and (ib (k) , jb (k) ) 
in the coordinate system of the first image, with a maximum 
possible error equal to noise E with a magnitude gained up 

15 by a factor Mk . That is, the (absolute value of) difference 
between the integral of true blue light intensity Lb(x, y) 
and the observed pixel value Bobs(k, ib, jb) is equal to or 
less than a maximum possible value of noise. Note that in 
formula (22), |x| denotes the absolute value of x. 

20 Now, referring to a flow chart shown in Fig. 6, a 

correct image estimation process in step S4 shown in Fig. 2 
according to a first embodiment of the present invention is 
described . 




Lb(x, y)dxdy-Bobs(k, ib, jb) < Mk xE 



• • • (22) 



5 



Formula (22) represents that the pixel value Bobs(k, ib, 



First, in step Sll, the calculation circuit 24 produces 
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a light intensity integral formula in the form (9) from the 
pixel value Gobs(l, ig, jg) of each pixel of the first image 
for each (ig, jg) , a light intensity integral formula in the 
form (10) from Robs(l, ir, jr)for each (ir, jr), and a light 
5 intensity integral formula in the form (11) from Bobsd, ib, 
jb) for each (ib, jb) . Thereafter, the process proceeds to 
step S12 . 

In step S12, the calculation circuit 24 produces a 
light intensity integral formula in the form (17) from the 

10 pixel value Gobs (2, ig, jg) of each pixel of the second 

image for each (ig, jg) , a light intensity integral formula 
in the form (18) from Robs (2, ir, jr)for each (ir, jr), and 
a light intensity integral formula in the form (19) from 
Bobs(2, ib, jb) for each (ib, jb) . Thereafter, the process 

15 proceeds to step S13 . In the above process, the calculation 
circuit 24 converts positions of the second image to 
corresponding position of the first image by using the 
conversion parameters (02, T2x, T2y, S2) supplied from the 
motion detector 23-1. 

20 In step S13, the calculation circuit 24 produces a 

light intensity integral formula in the form (20) from the 
pixel value Gobs(k, ig, jg) of each pixel of the k-th (k = 3 
to N) image for each (ig, jg) , a light intensity integral 
formula in the form (21) from Robs(k, ir, jr)for each (ir, 

25 jr) , and a light intensity integral formula in the form (22) 
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from Bobs(k, ib, jb) for each (ib, jb) . Thereafter, the 
process proceeds to step S14 . In the above process, the 
calculation circuit 24 converts positions of the k-th image 
to corresponding position of the first image by using the 
5 conversion parameters (0k, Tkx, Tky, Sk) supplied from the 
motion detector 23- (k - 1) . 

In step S14, the calculation circuit 24 estimates the 
true green light intensity Lg(x, y) , the true red light 
intensity Lr(x, y) and the true blue light intensity Lb(x, 

10 y) by solving formulas associated with the light density 

integral constraints (9) , (10) , (11) , (17) , (18) , (19) , (20) , 
(21), and (22) produced in steps Sll to S13 . If these 
values have been determined, the process is exited. The 
obtained values of the true green light intensity Lg(x, y) , 

15 the true red light intensity Lr(x, y) and the true blue 
light intensity Lb(x, y) (estimated values thereof) are 
supplied as resultant image signals (a G signal, a R signal, 
and B signal) representing a sharp image to the D/A 
converter 9 or the codec 12 . 

20 In this step S14, when the formulas associated with the 

light intensity integral constraint are solved to determine 
the true green light intensity Lg(x, y) , the true red light 
intensity Lr(x, y) and true the blue light intensity Lb(x, 
y) , a method of determining image data that satisfies a 

25 plurality of constraints is used. A specific example of 
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such a method is a POCS (Projection Onto Convex Sets) method. 
In the POCS method, a maximum- likelihood solution that 
satisfies given constraints is estimated by repeatedly 
performing projection onto convex sets. A description of 
5 the details of this method may be found, for example, in a 
paper "Image Restoration by the Method of Convex 
Projections: part 1 theory" (D. C. Youla and H. Webb, IEEE 
Trans. Med. Image., vol. 1, No. 2, pp81-94, Oct., 19 82). 
Japanese Unexamined Patent Application Publication No. 8- 

10 263639 discloses a technique of producing high-resolution 

still image from a plurality of low-resolution motion images 
by using the POCS method. In this technique disclosed in 
Japanese Unexamined Patent Application Publication No. 8- 
263639, motion compensation is performed using motion 

15 vectors, and identification accuracy is equal to one or 1/2 
pixels. The number of pixels used to estimate a most- 
likelihood pixel value of each pixel is fixed depending on 
the identification accuracy. In contrast, in the present 
embodiment, motion compensation is performed based on a 

20 semi -analog process, and the number of pixels used to 

estimate a most-likelihood pixel value of each pixel is 
adaptively changed depending on the state associated with 
the displacement due to the camera shake. 

Now, referring to a flow chart shown in Fig. 7, a 

25 correct image estimation processing step S4 shown in Fig. 2 
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according to the first embodiment of the present invention 
is further described. 

First, in step S21, the calculation circuit 24 sets a 
variable k indicating the count of the number of images 
5 stored in the calculation circuit 24 to 1 . Thereafter, the 
process proceeds to step S22 . 

In step S22, the calculation circuit 24 produces a 
light intensity integral formula in the form (20) from the 
pixel value Gobs(k, ig, jg) of each pixel of the k-th image 
10 for each (ig, jg) . Thereafter, the process proceeds to step 

523 . 

In step S23, the calculation circuit 24 produces a 
light intensity integral formula in the form (21) from the 
pixel value Robs(k, ir, jr) of each pixel of the k-th image 
15 for each (ir, jr) . Thereafter, the process proceeds to step 

524 . 

In step S24, the calculation circuit 24 produces a 
light intensity integral formula in the form (22) from the 
pixel value Bobs(k, ib, jb) of each pixel of the k-th image 
20 for each (ib, jb) . Thereafter, the process proceeds to step 

525 . 

When the inequalities associated with the light 
intensity integral are produced in steps S22 to S24 
described above, the conversion parameters supplied from the 
25 motion detectors 23-1 to 23- (N - 1) are used as required. 
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In step S25, the calculation circuit 24 determines 
whether the internal variable k is equal to the number of 
images N supplied from the frame memories 22-1 to 22 -N. If 
it is determined that the variable k is not equal to the 
5 number of images N, the process proceeds to step S26 in 

which the calculation circuit 24 increments the variable k 
by 1. Thereafter, the calculation circuit 24 returns the 
processing flow to step S22 and repeats the process from S22 
to S25. 

10 On the other hand, if it is determined that the 

variable k is equal to the number of images N, the process 
proceeds to step S27 in which the calculation circuit 24 
estimates the true green light intensity Lg(x, y) , the true 
red light intensity Lr(x, y) and the true blue light 

15 intensity Lb(x, y) by solving inequalities (20), (21), and 
(22) produced in steps S22 to S24 . If these values have 
been determined, the process is completed. The obtained 
values of the green light intensity Lg(x, y) , the red light 
intensity Lr(x, y) and the blue light intensity Lb(x, y) 

20 that are likely to be true values are supplied as resultant 
image signals (a G signal, a R signal, and B signal) 
representing a sharp image to the D/A converter 9 or the 
codec 12 . 

The true green light intensity Lg(x, y) , the true red 
25 light intensity Lr(x, y) , and the true blue light intensity 
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Lb(x, y) obtained herein are analog signals. That is, Lg(x, 
y) , Lr(x, y) , and Lb(x, y) are functions of (x, y) wherein, 
as described earlier, variables x and y are not integers but 
real numbers that can include decimal fractions. In general, 
5 each decimal fraction has two or three digits in binary, 
although the number of digits depends on the precision of 
the apparatus. Because Lg(x, y) , Lr(x, y) , and Lb(x, y) are 
functions of real numbers (x, y) , resampling is performed at 
integral numbers of (x, y) as required, and the resultant 
10 digital image signal is supplied to the D/A converter 9 or 
the codec 12. 

In a case in which a subject moves during a period in 
which N images are taken at a high shutter speed, the true 
green light intensity Lg(x, y) , the true red light intensity 
15 Lr(x, y) , and the true blue light intensity Lb(x, y) of a 

moving part of the subject change with time. In such a case, 
the method described above does not provide a correct 
solution . 

Thus, when the subject includes a moving part, an 
20 exception handling process is performed. In the exception 
handling process, simple superposition is performed. More 
specifically, from Bayer-array image data of each N images 
(having only one of R, G, and B signals at each pixel) , 
three signals consisting of R, G, and B signals are restored 
25 for each pixel by means of a demosacing process. After the 
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demosacing process, alignment is performed by rotating, 
increasing or reducing in size, and/or parallel displacing N 
images such that the N images are brought to correct 
positions, and the N images are superimposed together. The 
5 demosacing process may be performed using any known 
technique . 

In the first embodiment, as described above, the 
processing is performed taking into account the 
characteristic of the on-chip lens disposed directly on each 
10 pixel, and thus a sharp displacement-compensated image can 
be produced . 

Now, a correct image estimation process performed by 
the signal processor 7 according to a second embodiment of 
the present invention is described. 

15 In the second embodiment, in addition to the constraint 

associated with the light intensity integral given by 
formulas (20) to (21) for k = 1 to N, a constraint 
associated with color correlation among R, G, and B signals 
is imposed on the process of determining the true green 

20 light Lg(x, y) , the true red light intensity, and the blue 
light intensity Lb(x, y) . 

In a local area of an image, there is a color 
correlation among colors of the true green light intensity 
Lg(x, y) / the true red light intensity Lr(x, y) , and the 

25 true blue light intensity Lb(x, y) corresponding to light 
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originating from a subject and being incident on the imaging 
device 4. By imposing the color correlation in addition to 
the constraint associated with the light intensity integral 
given in formulas (20) to (22) , it becomes possible to 
5 obtain a better solution, that is, a sharper image more 
likely to be a correct image. 

Referring to Figs. 8 and 9, a specific method of 
determining a color correlation constraint is described 
below. In the following description referring to Figs. 8 
10 and 9, the correlation between green and red colors will be 
discussed. 

By way of example, the discussion will be made for a 
green pixel G(jg - 1) (ig - 1) at a (ig-th, jg-th) position 
of a k ! -th image shown in a lower left part of Fig. 8 a red 

15 pixel G(jg - 1) (ig - 1) at a (ir-th, jr-th) position of a 
k tf -th image shown in a lower right part of Fig. 8. 

The calculation circuit 24 converts positions of the 
green pixel G(jg - l)(ig - 1) of the k'-th image and the red 
pixel R(jr - 1) (ir - 1) of the k''-th image into positions 

20 in the coordinate system of the first image on the upper 
side of Fig. 8 by using conversion parameters (0k 1 , Tk'x, 
Tk'y, Sk') and (9k' 1 , Tk''x, Tk''y, Sk 1 ' ) in a similar 
manner to that according to the first embodiment. Note that 
k' and k' 1 can be equal to 1. When k' or k ' ' is equal to 1, 

25 the conversion parameters are given as (01, Tlx, Tly, SI) = 
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(0, 0, 0, 1) . 

The calculation circuit 24 then calculates the distance 
between the green pixel G(jg - 1) (ig - 1) of the k'-th image 
at the position converted into the coordinate system of the 
first image and the red pixel R(jr - 1) (ir - 1) of the k 1 * - 
th image at the position converted into the coordinate 
system of the first image. Furthermore, the calculation 
circuit 24 determines whether these two pixels can be 
regarded as locating at the same position by determining 
whether the calculated distance is less than a maximum 
allowable value (threshold value) delta (set to, for example, 
0.25 times the pixel size) . 

When positions of pixels G(jg - 1) (ig - 1) and R(jr - 
1) (ir - 1) are respectively given as positions (ig, jg) and 
(ir, jr) , if the position (ig, jg) of the pixel G(jg - 1) (ig 
- 1) in the coordinate system of the k'-th image is denoted 
as point G( C j , the position (ir, jr) of the pixel R(jg - 
1) (ig - 1) in the coordinate system of the k' ' -th image is 
denoted as point R {c) , the position (ig, jg) of the pixel 
G(jg - 1) (ig - 1) in the coordinate system of the first 
image is denoted as G c < k .), and the position (ir, jr) of the 
pixel R(jg - 1) (ig - 1) in the coordinate system of the 
first image is denoted as R C (k), then a constraint requiring 
that the distance bet ween the point G C (k') and point Rctk'*) 
should be within an allowable value delta is given by 
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= Dis 



Sk' 



Sk' 




< delta 



formula (23) shown below. 

Dis[G c(n ,R c(n ] 

cos (9k 9 ) -sin(0IO 
v sin(0k') cos(0k'), 

/cos (9k") -sin (9k") 
[sin (9 k") cos(9k")J\jrJ \Jk"yJ 

• • • (23) 

Formula (23) is referred to as a distance constraint 
5 formula. In formula (23), Dis[G c(k .), R C (k'-)] denotes the 

distance between point G C (kM and point R c <k''>- The positions 
of point G C (k-)) anc * point R C (k'M are obtained by performing 
the affine transformation on positions (ig, jg) and (ir, jr) 
using conversion parameters (0k 1 , Tk'x, Tk'y, Sk 1 ) and (0k 1 ', 

10 Tk 1 'x, Tk' ! y, Sk 1 1 ) according to an equation similar to 
equations (1) to (3). 

The calculation circuit 24 determines whether there are 
a green pixel G(jg - 1) (ig - 1) of the k'-th image and a red 
pixel R(jr - 1) (ir - 1) of the k''-th image at positions 

15 that can be regarded as being the same within an allowable 
margin delta in a small region (x ± dX, y ± dY) centered at 
a position (x, y) in the coordinate system of the first 
image, that is, in a rectangular region whose corners are at 
(x - dX, y - dY) , (x - dX, y + dY) , (x + dX, y - dY) , and (x 

20 + dX, y + dY) , wherein dX and dY are values that define the 
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region in the vicinity of the position (x, y) and they are 
set to be equal to, for example, 2 times the size of a pixel 
in the X direction and Y direction, respectively. 

In other words, the calculation circuit 24 determines 
5 (k', ig, jg) and (k 11 , ir, jr) that satisfy formula (23) 
within the small region (x ± dX, y ± dY) centered at the 
position (x, y) in the coordinate system of the first image, 
that is, in the rectangular region whose corners are at (x - 
dX, y - dY) , (x - dX, y + dY) , (x + dX, y - dY) , and (x + dX, 
10 y + dY) . 

The calculation circuit 24 then acquires pixel values 
Gobs(k', ig, jg) and Robs(k'', ir, jr) respectively 
corresponding to (k 1 , ig, jg) and (k'', ir, jr) determined. 

The calculation circuit 24 determines (k', ig, jg) and 
15 (k'', ir, jr) that satisfy formula (23) for all possible 
combinations of k' and k 1 1 within the range form 1 to N. 

In general, a plurality of combinations of (k', ig, jg) 
and (k'*, ir, jr) is detected, and the calculation circuit 
24 plots pixel values Gobs(k', ig, jg) and Robs(k'', ir, jr) 
20 corresponding to the detected (k', ig, jg) and (k fl , ir, jr) 
in the GR space in which the horizontal axis represents the 
G signal (Gobs(k', ig, jg) ) and the vertical axis represents 
the R signal (Robs(k'', ir, jr)), as shown in Fig. 9. 

Fig. 9 is a schematic diagram showing pixel values 
25 Gobs(k', ig, jg) and Robs(k'', ir, jr) regarded as locating 
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at same positions and plotted in the GR space. 

In Fig. 9, each x-like mark indicates a pair of pixel 
values Gobs(k f , ig, jg) and Robs(k ,l / ir, jr) corresponding 
to (k', ig, jg) and (k' 1 , ir, jr) detected by the 
5 calculation circuit 24 . 

The plots shown in Fig. 9 indicate that there is a 
correlation between the true green light intensity Lg(x, y) 
and the true red light intensity Lr(x, y) to be determined 
in the vicinity of the position (x, y) . 

10 Thus, in the second embodiment, in addition to 

constraints associated with the light intensity integral 
given by formulas (20) to (22) in the first embodiment, the 
correlation between green and red colors shown in Fig. 9 is 
added as a constrain. 

15 The calculation circuit 24 performs principal component 

analysis for the plurality of points plotted in the GR space 
shown in Fig. 9, wherein each plotted point corresponds to a 
pair of pixel values Gobs(k', ig, jg) and Robs(k*', ir, jr) 
that the calculation circuit 24 has detected as satisfying 

20 the distance constraint (23) . 

The calculation circuit 24 then determines the variance 
of a component (for example, a second principal component) 
in a direction perpendicular to a direction (denoted by a 
great arrow in Fig. 9) of a principal component (first 

25 principal component) obtained as a result of the principal 
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component analysis. Furthermore, the calculation circuit 24 
expresses a color correlation constraint in the form of a 
mathematical expression asserting that, for the position (x, 
y) , a point corresponding to true green light intensity Lg(x, 
5 y) and true red light intensity Lr(x, y) should be within a 
band extending in the GR space in the direction of principal 
component and having a width equal to the variance of the 
component in the direction perpendicular to the direction of 
the principal component. 

10 The color correlation constraint between green and blue 

is also introduced in a similar manner as described below. 

As with the green and red pixels shown in Fig. 8, the 
calculation circuit 24 converts positions of a green pixel 
G(jg - 1) (ig - 1) of the k'-th image and a blue pixel B(jb - 

15 1) (jb - 1) of the k f 1 ' -th image into positions in the 

coordinate system of the first image by using conversion 
parameters Ok', Tk'x, Tk'y, Sk ' ) and (9k ' ! ', Tk'''x, Tk'''y, 
Sk' 1 ! ) in a similar manner to that according to the first 
embodiment . 

20 The calculation circuit 24 then calculates the distance 

between the green pixel G(jg - l)(ig - 1) of the k'-th image 
at the position converted into the coordinate system of the 
first image and the blue pixel V(jb - 1) (ib - 1) of the 
k ,lf -th image at the position converted into the coordinate 

25 system of the first image. Furthermore, the calculation 
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circuit 24 determines whether these two pixels can be 
regarded as locating at the same position by determining 
whether the calculated distance is less than a maximum 
allowable value (threshold value) delta in a similar manner 
5 to Fig. 8. 

When positions of pixels G(jg - 1) (ig - 1) and B(jb - 
1) (ib - 1) are respectively given as positions (ig, jg) and 
(ib # jb) , if the position (ig, jg) of the pixel G(jg - 1) (ig 
- 1) in the coordinate system of the k'-th image is denoted 

10 as G( C ), the position (ib, jb) of the pixel B(jb - 1) (ib - 1) 
in the coordinate system of the k' 1 ■ -th image is denoted as 
point B( C ), the position (ig, jg) of the pixel G(jg - 1) (ig - 
1) in the coordinate system of the first image is denoted as 
G C (k')/ an< 3 the position (ib, jb) of the pixel B(jb - 1) (ib - 

15 1) in the coordinate system of the first image is denoted as 
B C (K'")/ then a constraint requiring that the distance between 
point Gc(k') and point B c(k ...) should be within the allowable 
value delta is given by formula (24) shown below. 



Dis [ G c(k')' B c(k"')] 




sin(0k')\/ig 

008(010/1 jg 





20 



(24) 
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Formula (24) is referred to as a distance constraint 
formula. In formula (24), Dis[G c(k .), B c < k ...)] denotes the 
distance between point G c(k .) and point B c(k ...) . The positions 
of point G c(k .)) and point B c(k ...) are obtained by performing 
5 the affine transformation on positions (ig, jg) and (ib, jb) 
using conversion parameters (6k', Tk'x, Tk f y, Sk ' ) and 
(0k 1 ' 1 , Tk 1 • ! x, Tk 1 ' f y, Sk ' ' ' ) according to an equation 
similar to equations (1) to (3). 

The calculation circuit 24 determines whether there are 

10 a green pixel G(jg - 1) (ig - 1) of the k'-th image and a 
blue pixel B(jb - 1) (ib - 1) of the k' 1 ' -th image at 
positions that can be regarded as being the same within an 
allowable margin delta in a small region (x ± dX, y ± dY) 
centered at a position (x, y) in the coordinate system of 

15 the first image, that is, in a rectangular region whose 

corners are at (x - dX, y - dY) , (x - dX, y + dY) , (x + dX, 
y - dY) , and (x + dX, y + dY) wherein dX and dY are values 
that define the region in the vicinity of the position (x, 
y) and they are set to be equal to, for example, 2 times the 

20 size of a pixel in the X direction and Y direction, 
respectively . 

In other words, the calculation circuit 24 determines 
(k 1 , ig, jg) and (k 1 ', ib, jb) that satisfy formula (24) 
within the small region (x ± dX, y ± dY) centered at the 
25 position (x, y) in the coordinate system of the first image, 
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that is, in the rectangular region whose corners are at (x - 
dX, y - dY) , (x - dX, y + dY) , (x + dX, y - dY) , and (x + dX, 
y + dY) . 

The calculation circuit 24 then acquires pixel values 
5 Gobs(k', ig, jg) and Bobs(k''', ib, jb) respectively 

corresponding to (k 1 , ig, jg) and (k 11 ', ib, jb) determined. 

The calculation circuit 24 determines (k', ig, jg) and 
(k'', ib, jb) that satisfy formula (24) for all possible 
combinations of k 1 and k' 1 ' within the range form 1 to N. 
10 In general, a plurality of combinations of (k», ig, jg) 

and (k T!! , ib, jb) is detected, and the calculation circuit 
24 plots pixel values Gobs(k , / ig, jg) and Bobs(k ,lf , ib, 
jb) corresponding to the detected (k', ig, jg) and (k ,! ', ib, 
jb) in the GB space in which the horizontal axis represents 
15 the G signal (Gobs(k', ig, jg) ) and the vertical axis 
represents the B signal (Bobs(k ! ' ' , ib, jb) ) . 

Thus, in the second embodiment, in addition to 
constraints associated with the light intensity integral 
given by formulas (20) to (22) in the first embodiment, the 
20 correlation between green and blue colors similar to the 

correlation between green and red shown in Fig. 9 is added 
as a constrain. 

The calculation circuit 24 performs principal component 
analysis for the plurality of points plotted in the GB space, 
25 wherein each plotted point corresponds to a pair of pixel 
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values Gobs(k', ig, jg) and BobsOc''', ib, jb) detected by 
the calculation circuit 24. 

The calculation circuit 24 then determines the variance 
of a component (for example, a second principal component) 
5 in a direction perpendicular to a direction of a principal 
component (first principal component) obtained as a result 
of the principal component analysis. Furthermore, the 
calculation circuit 24 expresses a color correlation 
constraint in the form of a mathematical expression 

10 asserting that, for the position (x, y) , a point 

corresponding to true green light intensity Lg(x, y) and 
true blue light intensity Lb(x, y) should be within a band 
extending in the GB space in the direction of principal 
component and having a width equal to the variance of the 

15 component in the direction perpendicular to the direction of 
the principal component . 

Thus, the true green light intensity Lg(x, y) , the true 
red light intensity Lr(x, y) , and the true blue light 
intensity Lb(x, y) finally obtained by the calculation 

20 circuit 24 are limited to those satisfying the constraint 

that a point corresponding to the true green light intensity 
Lg(x, y) and the true red light intensity Lr(x, y) at the 
position (x, y) exists within a band extending, in the GR 
space, in the direction of the principal component and 

2 5 having a width equal to the variance of the component 
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perpendicular to the direction of the principal component, 
and satisfying the constraint that a point corresponding to 
the true green light intensity Lg(x, y) and the true blue 
light intensity Lb(x, y) at the position (x, y) exists 
5 within a band extending, in the GB space, in the direction 
of the principal component and having a width equal to the 
variance of the component perpendicular to the direction of 
the principal component. 

Although in the present embodiment, the constraint 

10 associated with the color correlation in the GR space and 

the constraint associated with the color correlation in the 
GB space, a similar constraint associated with a color 
correlation between the R signal and the B signal (in a RB 
space) may also be employed in addition to the above 

15 constraints. 

As in the first embodiment, for example, the POCS 
method may be used to determine the true green light 
intensity Lg(x, y) , the true red intensity Lr(x, y) , and the 
true blue light intensity Lb(x, y) that satisfy both the 

20 constraint associated with the light intensity integrals and 
the constraint associated with the color correlations. 

The color correlation constraints may be imposed on all 
positions (x, y) or only particular positions such as 
lattice points at which x and y are equal to integers. 

25 Now, referring to a flow chart shown in Fig. 10, a 



1 
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correct image estimation process in step S4 shown in Fig. 2 
according to a second embodiment of the present invention is 
described. 

Steps S31 to S33 are similar to steps Sll to S13 in the 
5 correct image estimation process according to the first 
embodiment shown in Fig. 6. 

That is, in step S31, the calculation circuit 24 
produces a light intensity integral formula in the form (9) 
from the pixel value Gobs(l, ig, jg) of each pixel of the 

10 first image for each (ig, jg) , a light intensity integral 
formula in the form (10) from Robs(l, ir, jr)for each (ir, 
jr) , and a light intensity integral formula in the form (11) 
from Bobs(l, ib, jb) for each (ib, jb) . Thereafter, the 
process proceeds to step S32. 

15 In step S32, the calculation circuit 24 produces a 

light intensity integral formula in the form (17) from the 
pixel value Gobs (2, ig, jg) of each pixel of the second 
image for each (ig, jg) , a light intensity integral formula 
in the form (18) from Robs (2, ir, jr)for each (ir, jr) , and 

20 a light intensity integral formula in the form (19) from 

Bobs(2, ib, jb) for each (ib, jb) . Thereafter, the process 
proceeds to step S33. In the above process, the calculation 
circuit 24 converts positions of the second image to 
corresponding position of the first image by using the 

25 conversion parameters (02, T2x, T2y, S2) supplied from the 
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motion detector 23-1. 

In step S33, the calculation circuit 24 produces a 
light intensity integral formula in the form (20) from the 
pixel value Gobs(k, ig, jg) of each pixel of the k-th (k = 3 
5 to N) image for each (ig, jg) , a light intensity integral 
formula in the form (21) from Robs(k, ir, jr)for each (ir, 
jr) , and a light intensity integral formula in the form (22) 
from Bobs(k, ib, jb) for each (ib, jb) . Thereafter, the 
process proceeds to step S34 . In the above process, the 

10 calculation circuit 24 converts positions of the k-th image 
to corresponding position of the first image by using the 
conversion parameters (0k, Tkx, Tky, Sk) supplied from the 
motion detector 23- (k - 1) . 

In step S34, for a position (x, y) of interest, the 

15 calculation circuit 24 determines a pair of (k 1 , ig, jg) and 
(k fl 7 ir, jr) that satisfy the distance constraint 
represented in equality (23) in the vicinity of the position 
(x, y) . Thereafter, the process proceeds to step S35. 

In step S35, the calculation circuit 24 plots, in the 

20 GR space, the pixel values Gobs(k', ig, jg) and Robs(k , ' / ir, 
jr) at respective positions (k 1 , ig, jg) and (k'', ir, jr) 
determined in step S34 and performs principal component 
analysis. The calculation circuit 24 then determines the 
variance in a direction perpendicular to a principal 

25 direction determined as a result of the principal component 
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analysis, and represents a color correlation constraint in 
the form of a mathematical expression asserting that the G 
signal and the R signal of the same pixel exist within the 
range equal to the determined variance. Thereafter, the 
5 process proceeds from step S35 to step S36. 

In step S36, for a position (x, y) of interest, the 
calculation circuit 24 determines a pair of (k', ig, jg) and 
(k ,! ', ib, jb) that satisfy the distance constraint 
represented in equality (24) in the vicinity of the position 

10 (x, y) . Thereafter, the process proceeds to step S37. 

In step S37, the calculation circuit 24 plots, in the 
GB space, the pixel values Gobs(k', ig, jg) and Bobs(k ,l? , 
ib, jb) at respective positions (k', ig, jg) and (k' 1 ', ib, 
jb) determined in step S3 6 and performs principal component 

15 analysis. The calculation circuit 24 then determines the 
variance in a direction perpendicular to a principal 
direction determined as a result of the principal component 
analysis, and represents a color correlation constraint in 
the form of a mathematical expression asserting that the G 

20 signal and the B signal of the same pixel exist within the 
range equal to the determined variance. Thereafter, the 
process proceeds from step S37 to step S38. 

In step S38, the calculation circuit 24 determines 
whether the color correlation constraint has been determined 

25 in the form of a mathematical expression for all positions 
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(x, y) . If it is determined in step S38 that the color 
correlation constraint has not been determined for all 
positions (x, y) , the process returns to step S34 . In step 
S34, the calculation circuit 24 selects a position (x, y) 
5 for which the color correlation constraint has not been 
determined, and repeats the process from S34 to S38. 

On the other hand, in the case in which it is 
determined in step S38 that the color correlation constraint 
has been determined for all positions (x, y) , the process 

10 proceeds to step S39. In step S39, the calculation circuit 
24 calculates the true green light density Lg(x, y) , the 
true red light density Lr(x, y) , and the true blue light 
density Lb(x, y) that satisfy all constraints determined in 
steps S31, S32, S33, S35, and S37. Thereafter, the 

15 processing flow returns from the current process. In the 

above calculation, the calculation circuit 24 determines the 
true green light intensity Lg(x, y) , the true red light 
intensity Lr(x, y) , and the true blue light intensity Lb(x, 
y) that satisfy all the light intensity constraint formulas 

20 determined in steps S31 to S33 and the color correlation 
constraints determined in steps S35 and S37. 

Now, referring to flow charts shown in Fig. 11 and 12, 
a correct image estimation process in step S4 shown in Fig. 
2 according to the second embodiment of the present 

25 invention is further described. 
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Steps S51 to S56 are similar to steps S21 to S26 in the 
correct image estimation process according to the first 
embodiment shown in Fig . 7 . 

That is, in step S51, the calculation circuit 24 sets a 
5 variable k indicating the count of the number of images 

stored in the calculation circuit 24 to 1. Thereafter, the 
process proceeds to step S52 . 

In step S52, the calculation circuit 24 produces a 
light intensity integral formula in the form (20) from the 
10 pixel value Gobs(k, ig, jg) of each pixel of the k-th image 
for each (ig, jg) . Thereafter, the process proceeds to step 

553 . 

In step S53, the calculation circuit 24 produces a 
light intensity integral formula in the form (21) from the 
15 pixel value Robs(k, ir, jr) of each pixel of the k-th image 
for each (ir, jr) . Thereafter, the process proceeds to step 

554 . 

In step S54, the calculation circuit 24 produces a 
light intensity integral formula in the form (22)' from the 
20 pixel value Bobs(k, ib, jb) of each pixel of the k-th image 
for each (ib, jb) . Thereafter, the process proceeds to step 

555 . 

When the inequalities associated with the light 
intensity integral are produced in steps S52 to S54 
25 described above, the conversion parameters supplied from the 
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motion detectors 23-1 to 23- (N - 1) are used as required. 

In step S55, the calculation circuit 24 determines 
whether the internal variable k is equal to the number of 
images N supplied from the frame memories 22-1 to 22 -N. If 
5 it is determined that the variable k is not equal to the 
number of images N, the process proceeds to step S56 in 
which the calculation circuit 24 increments the variable k 
by 1. Thereafter, the calculation circuit 24 returns the 
processing flow to step S52 and repeats the process from S52 
10 to S56. 

On the other hand, if it is determined in step S55 that 
the variable k is equal to the number of images N, the 
process proceeds to step S57. Steps S57 to S62 are similar 
to steps S34 to S39 shown in Fig. 10. 

15 That is, in step S57, for a position (x, y) of interest, 

the calculation circuit 24 determines a pair of (k', ig, jg) 
and (k' 1 , ir, jr) that satisfy the distance constraint 
represented in equality (23) in the vicinity of the position 
(x, y) . Thereafter, the process proceeds to step S58. 

20 In step S58, the calculation circuit 24 plots, in the 

GR space, the pixel values Gobs(k ! , ig, jg) and Robsfk 1 ', ir, 
jr) at respective positions (k*, ig, jg) and (k 1 ', ir, jr) 
determined in step S57 and performs principal component 
analysis. The calculation circuit 24 then determines the 

25 variance in a direction perpendicular to a principal 
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direction determined as a result of the principal component 
analysis, and represents a color correlation constraint in 
the form of a mathematical expression asserting that the G 
signal and the R signal of the same pixel exist within the 
5 range equal to the determined variance. Thereafter, the 
process proceeds from step S58 to step S59. 

In step S59, for a position (x, y) of interest, the 
calculation circuit 24 determines a pair of (k', ig, jg) and 
(k ,fl , ib, jb) that satisfy the distance constraint 

10 represented in equality (24) in the vicinity of the position 
(x, y) . Thereafter, the process proceeds to step S60. 

In step S60, the calculation circuit 24 plots, in the 
GB space, the pixel values Gobs(k', ig, jg) and BobsCk 11 ', 
ib, jb) at respective positions (k', ig, jg) and (k ,,! , ib, 

15 jb) determined in step S59 and performs principal component 
analysis. The calculation circuit 24 then determines the 
variance in a direction perpendicular to a principal 
direction determined as a result of the principal component 
analysis, and represents a color correlation constraint in 

20 the form of a mathematical expression asserting that the G 
signal and the B signal of the same pixel exist within the 
range equal to the determined variance. Thereafter, the 
process proceeds from step S60 to step S61. 

In step S61, the calculation circuit 24 determines 

25 whether the color correlation constraint has been determined 
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in the form of a mathematical expression for all positions 
(x, y) . If it is determined in step S61 that the color 
correlation constraint has not been determined for all 
positions (x, y) , the process returns to step S57. In step 
5 S57, the calculation circuit 24 selects a position (x, y) 
for which the color correlation constraint has not been 
determined, and repeats the process from S57 to S61. 

On the other hand, in the case in which it is 
determined in step S61 that the color correlation constraint 

10 has been determined for all positions (x, y) , the process 

proceeds to step S62 . In step S62, the calculation circuit 
24 calculates the true green light density Lg(x, y) , the 
true red light density Lr(x, y) , and the true blue light 
density Lb(x, y) that satisfy all constraints determined in 

15 steps S52, S53, S54 , S58, and S60. Thereafter, the 

processing flow returns from the current process. In the 
above calculation, the calculation circuit 24 determines the 
true green light intensity Lg(x, y) , the true red light 
intensity Lr(x, y) , and the true blue light intensity Lb(x, 

20 y) that satisfy all the light intensity constraint formulas 
determined in steps S52 to S54 and the color correlation 
constraints determined in steps S58 and S60. 

In the present embodiment, as described above, the 
constraint associated with correlation among the R signal, 

25 the G signal and the B signal is added to the constraints 
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associated with light intensity integrals given by- 
inequalities (20) , (21) , and (22) for k = 1 to N in the 
first embodiment, and the true green light intensity Lg(x, 
y) , the true red light intensity Lr(x, y) , and the true blue 
5 light intensity Lb (x y) satisfying all above constraints. 
This makes it possible to obtain a sharp image close to an 
ideal image. 

Although in the example described above, the color 
correlation constraints are defined in the 2 -dimensional 

10 color spaces, the color correlation constraint may be 
defined in a 3 -dimensional color space. 

Now, a correct' image estimation process performed by 
the signal processor 7 according to a third embodiment of 
the present invention is described. 

15 In the first and second embodiments described above, 

the true green light intensity Lg(x, y) , the true red light 
intensity Lr(x, y) , and the true blue light intensity Lb(x, 
y) are estimated taking into account the accurately 
formulated characteristic of the on-chip lens disposed 

20 direction on each pixel of the imaging device 4, thereby 
obtaining a sharp image . 

In the third embodiment, the effects of the on-chip 
lens are ignored, and it is assumed that the intensity of 
light originating from a subject and being incident on a 

25 point (for example, the barycenter of each pixel) of each 
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pixel is detected by that pixel of the imaging device 4 
(that is, point -sampled data is obtained by each pixel) . 

Furthermore, as will be described in detail later, a 
spring model is used in this third embodiment. This makes 
5 it possible to reduce the computational complexity required 
to determine the true green light intensity Lg(x, y) , the 
true red light intensity Lr(x, y) , and the true blue light 
intensity Lb(x, y) , compared with the POCS method used in 
the first and second embodiments. The third embodiment also 

10 provides an advantage that it is possible to produce a 

compensated image with sharpness similar to that achieved by 
the first or second embodiment and better than can be 
achieved by the conventional techniques. 

In the third embodiment, as described above, data 

15 sensed by each pixel is regarded as point-sampled data. 

More specifically, at a pixel at a (i-th, j-th) position of 
the imaging device 4 shown in Fig. 3, data is regarded as 
being sensed at a central point (with coordinates (i - 0.5, 
j - 0.5)) of the pixel at the (i-th, j-th) position. 

20 Therefore, for example, a pixel value Gobs(l, ig, jg) 

obtained as data by a pixel at a (ig-th, jg-th) position of 
a first image is green light intensity Lg(ig - 0.5, jg - 
0.5) sensed at a position (ig - 0.5, jg - 0.5) in the 
coordinate system of the first image. Similarly, a pixel 

25 value Robs(l, ir, jr) obtained as data by a pixel at a (ir- 
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th, jr-th) position of the first image is red light 
intensity Lr(ir - 0.5, jr - 0.5) sensed at a position (ir - 
0.5, jr - 0.5) in the coordinate system of the first image, 
and a pixel value Bobs(l, ib, jb) obtained as data by a 
5 pixel at a (ib-th, jb-th) position of the first image is 
blue light intensity Lb(ib - 0.5, jb - 0.5) sensed at a 
position (ib - 0.5, jb - 0.5) in the coordinate system of 
the first image. 

Fig. 13 shows the first image stored in the frame 

10 memory 22-1. 

In Fig. 13, as for the pixel G(jg - 1) (ig - 1) that 
senses a green component, a pixel value Gobsd, ig, jg) is 
observed at a position denoted by a solid circle. As for 
the pixel R(jr - l)(ir - 1) that senses a red component, a 

15 pixel value Robs(l, ir, jr) is observed at a position 

denoted by a solid square. As for the pixel B(jb - 1) (ib - 
1) that senses a blue component, a pixel value Bobs(l, ib, 
jb) is observed at a position denoted by a solid triangle. 
As described above, the pixel value of each pixel of the 

20 first image is observed at the center (barycenter) , as 

represented in the coordinate system of the first image, of 
each pixel. For example, in the case of a pixel at a (i-th, 
j-th) position, the pixel value is observed at a position (i 
- 0.5, j - 0.5). 

2 5 Fig. 14 shows a second image stored in the frame memory 
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22-2 . 

In Fig. 14, as for the pixel G(jg - 1) (ig - 1) that 
senses a green component, a pixel value Gobs (2, ig, jg) is 
observed at a position denoted by a solid circle. As for 
5 the pixel R(jr - l)(ir - 1) that senses a red component, a 
pixel value Robs (2, ir, jr) is observed at a position 
denoted by a solid square. As for the pixel B(jb - 1) (ib - 
1) that senses a blue component, a pixel value Bobs (2, ib, 
jb) is observed at a position denoted by a solid triangle. 

10 As described above, the pixel value of each pixel of the 
second image is observed at the center (barycenter) , as 
represented in the coordinate system of the second image, of 
each pixel. For example, in the case of a pixel at a (i-th, 
j-th) position, the pixel value is observed at a position (i 

15 - 0.5, j - 0.5) . 

As in the first and second embodiments, points on the 
second image are converted to corresponding points in the 
coordinate system of the first image. More specifically, 
each point on the second image is converted to a 

20 corresponding point in the coordinate system of the first 
image by using conversion parameters (92, T2x, T2y, S2) 
detected by the motion detector 23-1. 

Corresponding points in the coordinate system of the 
first image converted from points on the second image are 

25 also shown in Fig. 14, for pixel values Gobs(2, ig, jg) , 
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Robs (2, ir, jr), and Bobs (2, ib, jb) . 

That is, the pixel value Gobs (2, ig, jg) of a pixel 
G (jg ~ D(ig - 1) in the coordinate system of the second 
image is green light intensity Lg(x, y) observed at a 
5 position ( (ig - 0.5) < 2 > / (jg - 0.5) (2 ) ) in the coordinate 

system of the first image obtained by converting a position 
(ig - 0.5, jg - 0.5) in the coordinate system of the second 
image using conversion parameters (02, T2x, T2y, S2) 
detected by the motion detector 23-1. In Fig. 14, the 

10 position ( (ig - 0.5) (2 ) , (jg - 0.5) < 2 >) in the coordinate 

system of the first image where the pixel value Gobs (2, ig, 
jg) is indicated by an open circle. 

The pixel value Robs (2, ir, jr) of a pixel R(jr - 1) (ir 
- 1) in the coordinate system of the second image is red 

15 light intensity Lr(x, y) observed at a position ((ir - 

0.5) (2 ), (jr - 0.5) ( 2 )) in the coordinate system of the first 
image obtained by converting a position (ir - 0.5, jr - 0.5) 
in the coordinate system of the second image using 
conversion parameters (02, T2x, T2y, S2) detected by the 

20 motion detector 23-1. In Fig. 14, the position ( (ir - 

0.5) ( 2 ), (jr - 0.5) ( 2 )) in the coordinate system of the first 
image where the pixel value Robs (2, ir, jr) is indicated by 
an open square . 

The pixel value Bobs (2, ib, jb) of a pixel B(jb - 1) (ib 

25 - 1) in the coordinate system of the second image is blue 
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light intensity Lb(x, y) observed at a position ( (ib - 
0.5) ( 2 ), (jb - 0.5) ( 2 >) in the coordinate system of the first 
image obtained by converting a position (ib - 0.5, jb - 0.5) 
in the coordinate system of the second image using 
5 conversion parameters (92, T2x, T2y, S2) detected by the 
motion detector 23-1. In Fig. 14, the position ( (ib - 
0.5) ( 2 ), (jb - 0.5) ( 2 )) in the coordinate system of the first 
image where the pixel value Bobs (2, ib, jb) is indicated by 
an open triangle . 
10 Fig. 15 shows a third image stored in the frame memory 

22-3 . 

In Fig. 15, as for the pixel G(jg - 1) (ig - 1) that 
senses a green component, a pixel value Gobs (3, ig, jg) is 
observed at a position denoted by a solid circle. As for 

15 the pixel R(jr - l)(ir - 1) that senses a red component, a 
pixel value Robs (3, ir, jr) is observed at a position 
denoted by a solid square. As for the pixel B(jb - 1) (ib - 
1) that senses a blue component, a pixel value Bobs (3, ib, 
jb) is observed at a position denoted by a solid triangle. 

2 0 As described above, the pixel value of each pixel of the 
third image is observed at the center (barycenter) , as 
represented in the coordinate system of the third image, of 
each pixel. For example, in the case of a pixel at a (i-th, 
j-th) position, the pixel value is observed at a position (i 

25 - 0.5, j - 0.5) . 



U u. 
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As in the first and second embodiments, points on the 
third image are converted to corresponding points in the 
coordinate system of the first image. More specifically, 
each point on the third image is converted to a 
5 corresponding point in the coordinate system of the first 
image by using conversion parameters (03, T3x, T3y, S3) 
detected by the motion detector 23-2. 

Corresponding points in the coordinate system of the 
first image converted from points on the third image are 

10 also shown in Fig. 15, for pixel values Gobs(3, ig, jg) , 
Robs (3, ir, jr), and Bobs (3, ib, jb) . 

That is, the pixel value Gobs (3, ig, jg) of a pixel 
G(jg - 1) (ig - 1) in the coordinate system of the third 
image is green light intensity Lg(x, y) observed at a 

15 position ((ig - 0.5) (3) , (jg - 0.5) (3) ) in the coordinate 

system of the first image obtained by converting a position 
(ig - 0.5, jg - 0.5) in the coordinate system of the third 
image using conversion parameters (93, T3x, T3y, S3) 
detected by the motion detector 23-2. In Fig. 15, the 

20 position ( (ig - 0.5) (3) , (jg - 0.5) (3) ) in the coordinate 

system of the first image where the pixel value Gobs (3, ig, 
jg) is indicated by an open circle. 

The pixel value Robs (3, ir, jr) of a pixel R(jr - 1) (ir 
- 1) in the coordinate system of the second image is red 

25 light intensity Lr(x, y) observed at a position ( (ir - 
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0.5) ( 3 ), (jr - 0.5) (3) ) in the coordinate system of the first 
image obtained by converting a position (ir - 0.5, jr - 0.5) 
in the coordinate system of the third image using conversion 
parameters (63 , T3x, T3y, S3) detected by the motion 
5 detector 23-2. In Fig. 15, the position ( (ir - 0.5) < 3) , (jr 

- 0.5) ( 3 )) in the coordinate system of the first image where 
the pixel value Robs (3, ir, jr) is indicated by an open 
square . 

The pixel value Bobs (3, ib, jb) of a pixel B(jb - 1) (ib 
10 - 1) in the coordinate system of the third image is blue 
light intensity Lb(x, y) observed at a position ((ib - 
0.5) ( 3 ), (jb - 0.5) ( 3) ) in the coordinate system of the first 
image obtained by converting a position (ib-0.5, jb - 0.5) 
in the coordinate system of the third image using conversion 
15 parameters (63, T3x, T3y, S3) detected by the motion 

detector 23-2. In Fig. 15, the position ( (ib - 0.5) (3) , (jb 

- 0.5) (3) ) in the coordinate system of the first image where 
the pixel value Bobs (3, ib, jb) is indicated by an open 
triangle . 

20 Fig. 16 shows positions, converted into the coordinate 

system of the first image, at which pixel values Gobs(l, ig, 
jg) to Gobs(N, ig, jg) of respective first to N-th images 
are observed at a pixel G(jg - 1) (ig - 1) that senses a 
green component . 

25 In the example shown in Fig. 16, a pixel value Gobsd, 



- 94 - 

S04P0831 

i<J/ jg) of the first image is observed at the center (pixel 
center) of a pixel G(jg - 1) (ig - 1) at a (ig-th, jg-th) 
position in the coordinate system of the first image. A 
pixel value Gobs (2, ig , jg) of the second image is observed 
5 at a position converted into the coordinate system of the 
first image in an upper-left area of the pixel G(jg - 1) (ig 
- 1) . A pixel value Gobs (3, ig, jg) of the third image is 
observed at a position converted into the coordinate system 
of the first image in a lower-left area of the pixel G(jg - 

10 1) (ig - 1) . A pixel value Gobs (4, ig, jg) of the fourth 
image is observed at a position converted into the 
coordinate system of the first image in an upper-right area 
of the pixel G(jg - 1) (ig - 1) . Note that pixel values 
Gobs(k, ig, jg) (k = 5 to N) of fifth to N-th images are not 

15 shown in Fig. 16. 

In this third embodiment, the calculation circuit 24 
determines the true green light intensity Lg(ig - 0.5, jg - 
0.5) at a position (ig - 0.5, jg - 0.5) in the coordinate 
system of the first image, based on pixel values Gobs(k, ig, 

20 jg) (k = 1 to N) of respective first to N-th images. 

Herein, the center position (i - 0.5, j - 0.5) of a 
pixel at a (i-th, j-th) position in the coordinate system of 
the first image used as a reference image is rewritten as 
(I » , J» ) . 

25 That is, I' = i - 0.5 and J' = j - 0.5. The true green 
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light intensity Lg ( I 1 , J') to be determined can be 
represented by Lg(ig - 0.5, jg - 0.5) at the center position 
(ig - 0.5, jg - 0.5) of a pixel that senses a green 
component. Similarly, the true red light intensity Lr(I', 
5 J') to be determined can be represented by Lr(ir - 0.5, jr - 
0.5) at the center position (ir - 0.5, jr - 0.5) of a pixel 
that senses a red component, and the true blue light 
intensity Lb(I', J') to be determined can be represented by 
Lb(ib - 0.5, jb - 0.5) at the center position (ib - 0.5, jb 

10 - 0.5) of a pixel that senses a blue component. 

Fig. 17 shows positions, as represented in the 
coordinate system of the first image, of the true red light 
intensity Lr(I ! , J'), the true red light intensity Lr (I ' , 
J'), and the true blue light intensity Lb ( I ' , J') to be 

15 determined by the calculation circuit 24 as image signals of 
a sharp image . 

That is, in Fig. 17, the center positions (I 1 , J 1 ), as 
represented in the coordinate system of the first image, of 
respective pixels of the imaging device 4 at which the true 

20 green light intensity Lg ( I ' , J'), the true red light 

intensity Lr(I', J'), and the true blue light intensity 
Lb(r , J') to be determined are indicated by solid circles. 

Referring to Figs. 18 to 23, the spring model is 
described taking, as an example, a pixel that senses a green 

25 component. In Fig. 18 and following figures, the positions 
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(I', J'), as represented in the coordinate system of the 
first image, of the true green light intensity Lg(I', J')/ 
the true red light intensity Lr ( 1 1 , J 1 ), and the true blue 
light intensity Lb (I 1 , J') to be determined are indicated by 
5 solid circles, positions, as represented in the coordinate 
system of the first image, of observed pixel values Gobs(k, 
jg) / Robs(k, ir, jr) , Bobs(k, ib, jb) (k = 1 to N) are 
indicated by open circles. Hereinafter, the coordinate 
system of the first image used as the reference image will 

10 also be referred to as a reference coordinate system. 

In Fig. 18, a position (x, y) of a point A indicates a 
position ( (ig - 0.5) oo , (jg - 0.5) (k)) in the reference 
coordinate system obtained by converting a position (ig - 
0.5, jg " 0.5) at which a pixel value Gobs(k, ig, jg) of a 

15 pixel at a (ig-th, jg-th) position of a k-th image was 

observed, by using conversion parameters (0k, Tkx, Tky, Sk) 
detected by the motion detector 23- (k - 1) . When K = 1, the 
conversion parameters are regarded as (0k, Tkx, Tky, Sk) = 
(0, 0, 0, 1), and thus (x, y) = (ig - 0.5, jg - 0.5). Note 

20 that the pixel value at point A(x, y) is Gobs(k, ig, jg) . 

For the given point A(x, y) , integers I and J that 
satisfy I-0.5<x<l+0.5 and J-0.5<y<J+0.5, 
respectively, are determined. Positions J') at which 

to determine the true green light intensity Lg ( I ' , J 1 ) are 

25 given by four corners (I - 0.5, J - 0.5), (I - 0.5, J + 0.5), 
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(I + 0.5, J - 0.5), and (1+0.5, J+0.5) of a rectangle (I 
±0.5, J ± 0.5) . 

The pixel value Gobs(k, ig, jg) at the point A(x, y) 
can be regarded as being nearly equal to true green light 
5 intensities Lg(I - 0.5, J - 0.5), Lg(I - 0.5, J+ 0.5), Lg(I 
+ 0.5, J - 0.5), and Lg(I + 0.5, J + 0.5) at respectively 
positions (I - 0.5, J - 0.5), (I - 0.5, J + 0.5), (I + 0.5, 
J - 0.5), and (1+0.5, J+0.5) close to the point A(x, y) . 
That is, the true green light intensities Lg(I - 0.5, J - 

10 0.5), Lg(I - 0.5, J + 0.5), Lg ( I + 0.5, J - 0.5), and Lg ( I + 
0.5, J + 0.5) can be approximated by the pixel value Gobs(k, 
19/ jg) at the point A(x, y) . 

As described in the first and second embodiments, the 
pixel value Gobs(k, ig, jg) observed at the point A(x, y) 

15 includes an error (noise) Mk x E. Furthermore, each of the 
true green light intensities Lg(I - 0.5, J - 0.5), L.g(I - 
0.5, J + 0.5), Lg(I + 0.5, J -05), and Lg(I + 0.5, J + 0.5) 
at respective positions (I - 0.5, J - 0.5), (I - 0.5, J + 
0.5), (I + 0.5, J - 0.5), and (I + 0.5, J + 0.5) includes an 

20 additional error caused by approximating them by the pixel 
value Gobs(k, ig, jg) at the point A(x, y) . (Hereinafter, 
such an error will be referred to as an approximation error) . 

The approximation of the true green light intensity, 
for example, Lg(I - 0.5, J - 0.5) at the position (I - 0.5, 

25 J - 0.5) by the pixel value Gobs(k, ig, jg) at the point A(x, 
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y) can be represented by the spring model using a spring. 

Fig. 19 shows the concepts of the spring model by which 
the true green light intensity Lg(I - 0.5, J - 0.5) at the 
position (I - 0.5, J - 0.5) is approximated by the pixel 
5 value Gobs(k, ig, jg) at the point A(x, y) . 

As shown in Fig. 19, one end of a spring BA1 is 
connected to the pixel value Gobs(k, ig, jg) and the other 
end of the spring BA1 is connected to the true green light 
intensity Lg(I -0.5, J -0.5). In a particular state shown 

10 in Fig. 19, the true green light intensity Lg(I - 0.5, J - 

0.5) is being pulled toward the pixel value Gobs(k, ig, jg) . 
In the spring model shown in Fig. 19, the degree to which 
the true green light intensity Lg(I - 0.5, J - 0.5) is 
pulled toward the pixel value Gobs(k, ig, jg) increases with 

15 the likelihood that the pixel value Gobs(k, ig, jg) at the 
point A(x, y) is the true green light intensity Lg ( I - 0.5, 
J - 0.5). Conversely, the true green light intensity Lg ( I - 
0.5, J - 0.5) moves away from the pixel value Gobs(k, ig, 
jg) when the likelihood is low. 

2 0 The approximation error caused by approximating the 

true green light intensity Lg ( I - 0.5, J - 0.5) by the pixel 
value Gobs(k, ig, jg) increases with the distance between 
the point A(x, y) and the position (I - 0.5, J - 0.5). Thus, 
herein, a function F((xl, yl) , (x2, y2)) that increases with 

25 the distance between a point (xl, yl) and a point (x2, y2) 



- 99 - 

S04P0831 

is introduced. As for the function F((xl, yl) , (x2, y2)), 
for example, F((xl, yl) , (x2, y2 ) ) = V{ (xl - x2) 2 + (yl - 
y2) 2 } may be employed. 

As described earlier, the pixel value Gobs (k, ig, jg) 
5 observed at the point A(x, y) includes an error (noise) Mk x 
E caused by noise. 

The likelihood of equality between the true green light 
intensity Lg(I - 0.5, J - 0.5) and the pixel value Gobs (k, 

jg) / that is, the approximation accuracy of the true ^ 

10 green light intensity Lg(I - 0.5, J - 0.5) approximated by 
the pixel value Gobs(k, ig, jg) decreases or increases with 
increasing or decreasing approximation error and noise E x 
Mk. Therefore, the approximation accuracy (likelihood) 
decreases with (in inverse proportion to) Mk x E x F((x, y) , 

15 (I - 0.5, J - 0.5)). That is, the likelihood that the true 
green light intensity Lg(I - 0.5, J - 0.5) is equal to the 
pixel value Gobs(k, ig, jg) at the point A(x, y) is high 
when the value of Mk x E x F((x, y), (I - 0.5, J - 0.5)) is 
large. Conversely, the likelihood low is high when the 

20 value of Mk x E x F((x, y) , ( I - 0 . 5 , J - 0 . 5 ) ) is small. 

If the likelihood of equality of the true green light 
intensity Lg(I - 0.5, J - 0.5) to the pixel value Gobs(k, ig, 
jg) is represented by the spring constant (strength) of the 
spring BA1, then the spring constant is given, for example, 

25 as {V2 - F((x, y) , (I - 0.5, J - 0.5))}/(Mk x E) . The 
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denominator (Mk x E) is equal to Mk times the noise E. 
Therefore, as noise increases, the strength of the spring 
BA1 decreases and thus the force of pulling the true green 
light intensity Lg(I - 0.5, J - 0.5) toward the pixel value 
5 Gobs(k, ig, jg) at the point A decreases. The numerator {V2 

- F((x, y) , (I - 0.5, J - 0.5))} is equal to the value 
(difference) obtained by subtracting the distance between 
the point A and the point (I - 0.5, J - 0.5) from V (l 2 + l 2 ) 
= V2, that is, the maximum possible value of the distance 

10 between two arbitrary points within the rectangular region 
(I ±0.5, J ± 0.5) shown in Fig. 18. Therefore, as the 
distance increases between the point A(x, y) at which the 
pixel value Gobs(k, ig, jg) is observed and the position (I 

- 0.5, J - 0.5) of the true green light intensity Lg(I - 0.5, 
15 J - 0.5) to be determined, V2 - F((x, y), (I - 0.5, J - 

0.5)) decreases and thus the strength of the spring BA1 
decreases. Thus, the force of the spring BA1 to pull the 
true green light intensity Lg(I - 0.5, J - 0.5) toward the 
pixel value Gobs(k, ig, jg) at the point A decreases. 

20 Instead of {V2 - F((x, y) , (I - 0.5, J - 0.5))}, other 

functions such as the inverse of F((x, y) , (1-0.5, J - 
0.5)) may be used as the numerator of the spring constant. 

In the spring model, if the spring BA1 is assumed to 
have a natural length of 0 (that is, the length is equal to 

25 0 when no load is imposed on the spring) , then the length 
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(extension) of the spring BA1 along an axis of light 
intensity (pixel value) can be written as | Gobs(k, ig, jg) 

- Lg(I - 0.5, J - 0.5) |, and the force exerted by the 
spring BA1 on the true green light intensity Lg(I - 0.5, J - 

5 0.5) to pull it toward the pixel value Gobs(k, ig, jg) at 
the point A (x, y) can be written as the spring constant x 
the length of the spring = {V2 - F ( (x, y) , (I - 0.5, J - 
0.5))}/(Mk x E) } x | Gobs(k, ig, jg) - Lg(I - 0.5, J - 0.5) 

I- 

10 Fig. 20 shows the concepts of the spring model of the 

relationship between the pixel value Gobs (k, ig, jg) at the 
point A(x, y) and true green light intensities Lg(I - 0.5, J 

- 0.5), Lg ( I - 0.5, J + 0.5), Lg ( I + 0.5, J - 0.5), and Lg ( I 
+ 0.5, J + 0.5) at respective positions (I - 0.5, J - 0.5), 

15 (1-0.5, J + 0. 5), (1+0.5, J- 0.5), and (1+0.5, J + 
0.5). 

As with the spring model of the relationship between 
the pixel value Gobs(k, ig, jg) at the point A(x, y) and the 
true green light intensity Lg ( I - 0.5, J - 0.5) described 

20 above with reference to Fig. 19, the relationship of the 
true green light intensities Lg ( I - 0.5, J + 0.5), Lg(I + 
0.5, J - 0.5), and Lg ( I + 0.5, J + 0.5) with the pixel value 
Gobs(k, ig, jg) at the point A(x, y) is also represented by 
the spring model. 

2 5 As shown in Fig. 20, one end of a spring BA2 is 
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connected to the pixel value Gobs(k, ig, jg) and the other 
end of the spring BA2 is connected to the true green light 
intensity Lg(I - 0.5, J + 0.5), and the true green light 
intensity Lg ( I - 0.5, J + 0.5) is being pulled toward the 
5 pixel value Gobs(k, ig, jg) . In the spring model shown in 
Fig. 20, the degree to which the true green light intensity 
Lg(I - 0.5, J + 0.5) is pulled toward the pixel value Gobs(k, 
ig/ jg) increases with the likelihood that the pixel value 
Gobs(k, ig, jg) at the point A(x, y) is the true green light 

10 intensity Lg(I - 0.5, J + 0.5). Conversely, the true green 
light intensity Lg(I - 0.5, J + 0.5) moves away from the 
pixel value Gobs (k, ig, jg) when the likelihood is low. 

The approximation error caused by approximating the 
true green light intensity Lg(I - 0.5, J + 0.5) by the pixel 

15 value Gobs(k, ig, jg) increases with the distance between 

the point A(x, y) and the position (I - 0.5, J + 0.5). Thus, 
herein, a function F((xl, yl) , (x2, y2 ) ) that increases with 
the distance between a point (xl, yl) and a point (x2, y2) 
is introduced. As for the function F((xl, yl), (x2, y2) ) , 

20 for example, F((xl, yl) , (x2, y2)) = V{(xl - x2) 2 + (yl - 
y2) 2 } may be employed. 

As described earlier, the pixel value Gobs(k, ig, jg) 
observed at the point A(x, y) includes an error (noise) Mk x 
E caused by noise. 

25 The likelihood of equality between the true green light 
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intensity Lg(I - 0.5, J + 0.5) and the pixel value Gobs(k, 
ig* jg) / that is, the approximation accuracy of the true 
green light intensity Lg(I - 0.5, J + 0.5) approximated by 
the pixel value Gobs(k, ig, jg) decreases or increases with 
5 increasing or decreasing approximation error and noise E x 
Mk. Therefore, the approximation accuracy (likelihood) 
decreases with (in inverse proportion to) Mk x E x F ( (x, y) , 
(I - 0.5, J + 0.5)). That is, the likelihood that the true 
green light intensity Lg(I - 0.5, J + 0.5) is equal to the 

10 pixel value Gobs(k, ig, jg) at the point A(x, y) is high 

when the value of Mk x E x F((x, y), (I - 0.5, J + 0.5)) is 
large. Conversely, the likelihood low is high when the 
value of Mk x E x F((x, y) , (I - 0.5, J +0.5)) is small. 

If the likelihood of equality of the true green light 

15 intensity Lg(I - 0.5, J + 0.5) to the pixel value Gobs(k, ig, 
jg) is represented by the spring constant (strength) of the 
spring BA2 , then the spring constant is given, for example, 
as {V2 - F((x, y) , (I - 0.5, J + 0.5))}/(Mk x E) . The 
denominator (Mk x E) is equal to Mk times the noise E. 

20 Therefore, as noise increases, the strength of the spring 
BA2 decreases and thus the force of pulling the true green 
light intensity Lg(I - 0.5, J + 0.5) toward the pixel value 
Gobs(k, ig, jg) at the point A decreases. The numerator {V2 
- F((x, y) , (I - 0.5, J + 0.5))} is equal to the value 

25 (difference) obtained by subtracting the' distance between 
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the point A and the point (I - 0.5, J + 0.5) from V (l 2 + l 2 ) 
= V2, that is, the maximum possible value of the distance 
between two arbitrary points within the rectangular region 
(I ±0.5, J ± 0.5) shown in Fig. 18. Therefore, as the 
5 distance increases between the point A(x, y) at which the 
pixel value Gobs(k, ig, jg) is observed and the position (I 

- 0.5, J+ 0.5) of the true green light intensity Lg ( I - 0.5, 
J + 0.5) to be determined, V2 - F((x, y) , (I - 0.5, J + 
0.5)) decreases and thus the strength of the spring BA2 

10 decreases. Thus, the force of the spring BA2 to pull the 
true green light intensity Lg(I - 0.5, J + 0.5) toward the 
pixel value Gobs(k, ig, jg) at the point A decreases. 
Instead of {V2 - F((x, y) , (I - 0.5, J + 0.5))}, other 
functions such as the inverse of F((x, y) , (I - 0.5, J + 

15 0.5)) may be used as the numerator of the spring constant. 

In the spring model, if the spring BA2 is assumed to 
have a natural length of 0 (that is, the length is equal to 
0 when no load is imposed on the spring) , then the length 
(extension) of the spring BA2 along an axis of light 

20 intensity (pixel value) can be written as ) Gobs(k, ig, jg) 

- Lg(I - 0.5, J + 0.5) |, and the force exerted by the 
spring BA2 on the true green light intensity Lg(I - 0.5, J + 
0.5) to pull it toward the pixel value Gobs(k, ig, jg) at 
the point A (x, y) can be written as the spring constant x 

25 the length of the spring = {V2 - F((x, y) , (I - 0.5, J + 
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0.5))}/(Mk x E) } x | Gobs(k, ig, jg) - Lg(I - 0.5, J + 0.5) 
I • 

Furthermore, as shown in Fig. 20, one end of a spring 
BA3 is connected to the pixel value Gobs(k, ig, jg) and the 
5 other end of the spring BA3 is connected to the true green 
light intensity Lg(I + 0.5, J - 0.5), and the true green 
light intensity Lg(I + 0.5, J - 0.5) is being pulled toward 
the pixel value Gobs(k, ig, jg) . In the spring model shown 
in Fig. 20, the degree to which the true green light 

10 intensity Lg(I + 0.5, J - 0.5) is pulled toward the pixel 

value Gobs(k, ig, jg) increases with the likelihood that the 
pixel value Gobs(k, ig, jg) at the point A(x, y) is the true 
green light intensity Lg ( I + 0.5, J - 0.5). Conversely, the 
true green light intensity L»g(I + 0.5, J - 0.5) moves away 

15 from the pixel value Gobs(k, ig, jg) when the likelihood is 
low. 

The approximation error caused by approximating the 
true green light intensity Lg ( I + 0.5, J - 0.5) by the pixel 
value Gobs(k, ig, jg) increases with the distance between 

20 the point A(x, y) and the position (I + 0.5, J - 0.5). Thus, 
herein, a function F((xl, yl) , (x2, y2 ) ) that increases with 
the distance between a point (xl, yl) and a point (x2, y2) 
is introduced. As for the function F((xl, yl), (x2, y2)), 
for example, F((xl, yl) , (x2, y2)) = V{ (xl - x2) 2 + (yl - 

25 y2) 2 } may be employed. 
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As described earlier, the pixel value Gobs(k, ig, jg) 
observed at the point A(x, y) includes an error (noise) Mk x 
E caused by noise. 

The likelihood of equality between the true green light 
5 intensity Lg(I + 0.5, J - 0.5) and the pixel value Gobs(k, 
ig* j9) / that is, the approximation accuracy of the true 
green light intensity Lg(I + 0.5, J - 0.5) approximated by 
the pixel value Gobs(k, ig, jg) decreases or increases with 
increasing or decreasing approximation error and noise E x 

10 Mk. Therefore, the approximation accuracy (likelihood) 

decreases with (in inverse proportion to) Mk x E x F((x, y) , 
(I + 0.5, J - 0.5)). That is, the likelihood that the true 
green light intensity Lg(I + 0.5, J - 0.5) is equal to the 
pixel value Gobs(k, ig, jg) at the point A(x, y) is high 

15 when the value of Mk x E x F((x, y) , ( I + 0 . 5 , J - 0 . 5 ) ) is 
large. Conversely, the likelihood low is high when the 
value of Mk x E x F((x, y) , (1+0.5, J - 0.5)) is small. 

If the likelihood of equality of the true green light 
intensity Lg(I + 0.5, J - 0.5) to the pixel value Gobs(k, ig, 

20 jg) is represented by the spring constant (strength) of the 
spring BA3 , then the spring constant is given, for example, 
as {V2 - F((x, y) , (I + 0.5, J - 0.5))}/(Mk x E) . The 
denominator (Mk x E) is equal to Mk times the noise E. 
Therefore, as noise increases, the strength of the spring 

25 BA3 decreases and thus the force of pulling the true green 
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light intensity Lg(I + 0.5, J - 0.5) toward the pixel value 
Gobs(k, ig, jg) at the point A decreases. The numerator {^2 
- F((x, y) , (I + 0.5, J - 0.5))} is equal to the value 
(difference) obtained by subtracting the distance between 
5 the point A and the point (I + 0.5, J - 0.5) from V (l 2 + l 2 ) 
= V2, that is, the maximum possible value of the distance 
between two arbitrary points within the rectangular region 
(I ±0.5, J ±0.5) shown in Fig. 18. Therefore, as the 
distance increases between the point A(x, y) at which the 

10 pixel value Gobs(k, ig, jg) is observed and the position (I 

+ 0.5, J - 0.5) of the true green light intensity Lg(I + 0.5, 
J - 0.5) to be determined, V2 - F((x, y), (I + 0.5, J - 
0.5)) decreases and thus the strength of the spring BA3 
decreases. Thus, the force of the spring BA3 to pull the 

15 true green light intensity Lg(I + 0.5, J - 0.5) toward the 
pixel value Gobs(k, ig, jg) at the point A decreases. 
Instead of {V2 - F((x, y), (I + 0.5, J - 0.5))}, other 
functions such as the inverse of F((x, y) , (1+0.5, J - 
0.5)) may be used as the numerator of the spring constant. 

2 0 In the spring model, if the spring BA3 is assumed to 

have a natural length of 0 (that is, the length is equal to 
0 when no load is imposed on the spring) , then the length 
(extension) of the spring BA3 along an axis of light 
intensity (pixel value) can be written as j Gobs(k, ig, jg) 

25 - Lg(I + 0.5, J - 0.5) |, and the force exerted by the 
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spring BA3 on the true green light intensity Lg(I + 0.5, J - 
0.5) to pull it toward the pixel value Gobs(k, ig, jg) at 
the point A (x, y) can be written as the spring constant x 
the length of the spring = {V2 - F((x, y) , (I + 0.5, J - 
5 0.5))}/(Mk x E) } x I Gobs(k, ig, jg) - Lg ( I + 0.5, J - 0.5) 

I- 

Furthermore, in Fig. 20, one end of a spring BA4 is 
connected to the pixel value Gobs(k, ig, jg) and the other 
end of the spring BA4 is connected to the true green light 

10 intensity Lg(I + 0.5, J + 0.5), and the true green light 
intensity Lg(I + 0.5, J + 0.5) is being pulled toward the 
pixel value Gobs(k, ig, jg) . In the spring model shown in 
Fig. 20, the degree to which the true green light intensity 
Lg(I + 0.5, J + 0.5) is pulled toward the pixel value Gobs(k, 

15 ig, jg) increases with the likelihood that the pixel value 

Gobs(k, ig, jg) at the point A(x, y) is the true green light 
intensity Lg(I + 0.5, J + 0.5). Conversely, the true green 
light intensity Lg(I + 0.5, J + 0.5) moves away from the 
pixel value Gobs(k, ig, jg) when the likelihood is low. 

2 0 The approximation error caused by approximating the 

true green light intensity Lg(I + 0.5, J + 0.5) by the pixel 
value Gobs(k, ig, jg) increases with the distance between 
the point A(x, y) and the position (I + 0.5, J + 0.5). Thus, 
herein, a function F((xl, yl) , (x2, y2 ) ) that increases with 

25 the distance between a point (xl, yl) and a point (x2, y2) 
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is introduced. As for the function F((xl, yl) , (x2, y2) ) , 
for example, F((xl, yl) , (x2, y2)) = V{ (xl - x2) 2 + (yl - 
y2) 2 } may be employed. 

As described earlier, the pixel value Gobs(k, ig, jg) 
5 observed at the point A(x, y) includes an error (noise) Mk x 
E caused by noise. 

The likelihood of equality between the true green light 
intensity Lg(I + 0.5, J + 0.5) and the pixel value Gobs(k, 
19/ jg) / that is, the approximation accuracy of the true 

10 green light intensity Lg(I + 0.5, J + 0.5) approximated by 
the pixel value Gobs(k, ig, jg) decreases or increases with 
increasing or decreasing approximation error and noise E x 
Mk . Therefore, the approximation accuracy (likelihood) 
decreases with (in inverse proportion to) Mk x E x F((x, y) , 

15 (I + 0.5, J + 0.5)). That is, the likelihood that the true 
green light intensity Lg ( I + 0.5, J + 0.5) is equal to the 
pixel value Gobs(k, ig, jg) at the point A(x, y) is high 
when the value of Mk x E x F((x, y), (1+0.5, J + 0.5)) is 
large. Conversely, the likelihood low is high when the 

20 value of Mk x E x F((x, y) , (I + 0.5, J + 0.5)) is small. 

If the likelihood of equality of the true green light 
intensity Lg(I + 0.5, J + 0.5) to the pixel value Gobs (k, ig, 
jg) is represented by the spring constant (strength) of the 
spring BA4 , then the spring constant is given, for example, 

25 as {V2 - F((x, y) , (I + 0.5, J + 0.5))}/(Mk x E) . The 
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denominator (Mk x E) is equal to Mk times the noise E. 
Therefore, as noise increases, the strength of the spring 
BA4 decreases and thus the force of pulling the true green 
light intensity Lg(I + 0.5, J + 0.5) toward the pixel value 
5 Gobs(k, ig, jg) at the point A decreases. The numerator {V2 
- F((x, y) , (I + 0.5, J+ 0.5))} is equal to the value 
(difference) obtained by subtracting the distance between 
the point A and the point (I + 0.5, J + 0.5) from V (l 2 + l 2 ) 
= V2 , that is, the maximum possible value of the distance 

10 between two arbitrary points within the rectangular region 
(I ±0.5, J ± 0.5) shown in Fig. 18. Therefore, as the 
distance increases between the point A(x, y) at which the 
pixel value Gobs(k, ig, jg) is observed and the position (I 
+ 0.5, J + 0.5) of the true green light intensity Lg(I + 0.5, 

15 J + 0.5) to be determined, V2 - F((x, y) , (I + 0.5, J + 
0.5)) decreases and thus the strength of the spring BA4 
decreases. Thus, the force of the spring BA4 to pull the 
true green light intensity Lg(I + 0.5, J + 0.5) toward the 
pixel value Gobs(k, ig, jg) at the point A decreases. 

20 Instead of {V2 - F ( (x, y) , (I + 0.5, J + 0.5))}, other 

functions such as the inverse of F((x, y) , (I + 0.5, J + 
0.5)) may be used as the numerator of the spring constant. 

In the spring model, if the spring BA4 is assumed to 
have a natural length of 0 (that is, the length is equal to 

2 5 0 when no load is imposed on the spring) , then the length 
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(extension) of the spring BA4 along an axis of light 
intensity (pixel value) can be written as | Gobs(k, ig, jg) 
- Lg(I +0.5, J+0.5) |, and the force exerted by the 
spring BA4 on the true green light intensity Lg(I +0.5, J + 
5 0.5) to pull it toward the pixel value Gobs(k, ig, jg) at 
the point A (x, y) can be written as the spring constant x 
the length of the spring = {V2 - F((x, y) , (I + 0.5, J + 
0.5))}/(Mk x E) } x I Gobs(k, ig, jg) - Lg(I + 0.5, J + 0.5) 

I- 

10 In the process described above, for a given arbitrary 

position A(x, y) in the reference coordinate system, the 
spring model is introduced to express the relationship 
between the pixel value Gobs(k, ig, jg) at the position A(x, 
y) and the true green light intensities Lg(I - 0.5, J - 0.5), 

15 Lg(I - 0.5, J+ 0.5), Lg ( I + 0.5, J - 0.5), and Lg ( I + 0.5, 

J+0.5) at positions in the vicinity of the position (x, y) , 
that is, at positions indicated by integers I and J that 
satisfy inequalities I-0.5<x<I+0.5 and J - 0.5 < y 
<J + 0.5. Instead, for a given position (I 1 , J') of 

20 interest in the reference coordinate system at which the 

center of a pixel is located, the relationship between the 
true green light intensity Lg ( I ' , J') at the position (I 1 , 
J') and pixel values Gobs(k, ig, jg) observed in the 
vicinity of the position (I 1 , J 1 ) may be expressed by a 

25 spring model, as will be described below. 
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Note that (I 1 , J 1 ) indicates the center position of a 
pixel, and thus I', and J' have a value whose fractional 
portion is equal to 0.5. Therefore, in the specific example 
shown in Fig. 20, positions denoted by solid circles such as 
5 (1-0.5, J - 0.5) are regarded as positions (I 1 , J 1 ). 

As shown in Fig. 20, a spring model is defined for four 
true green light intensities Lg(I - 0.5, J - 0.5), Lg(I - 
0.5, J+ 0.5), Lg ( I + 0.5, J - 0.5), and Lg (I + 0.5, J + 
0.5) in the vicinity of the point A(x, y) . Similarly, in 

10 addition to the point A(x, y) , a spring model using four 
springs is defined for each point at which a pixel value 
Gobs(k, ig, jg) of first to N-th captured images is observed. 
Thus, for the position (I', J 1 ), as shown in Fig. 21, the 
spring model is defined so as to represent the relationship 

15 between the true green light intensity Lg ( I ' , J') at the 

position (I', J 1 ) and pixel values Gobs (k, ig, jg) observed, 
for example, at points A to E in the vicinity of the 
position (I ' , J' ) . 

That is, for the given position (I', J') of interest, 

20 all combinations of integral values of k, ig, and jg are 

determined which satisfy I l -l<x<I l +l and J ' - 1 < y 
< J 1 +1 with respect to the position (x, y) in the 
reference coordinate system obtained by converting, based on 
conversion parameters (6k, Tkx, Tky, Sk) , the position (ig - 

25 0.5, jg - 0.5). By way of example, let us assume that 
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combinations of integers (k, ig, jg) identifying respective 
five pixel values Gobs(k, ig, jg) observed at points A to E 
are obtained for the position (I', J') as shown in Fig. 21. 
In this case, as shown in Fig. 22, a spring model is 
5 defined to represent the relationship between the true green 
light intensity Lgd', J') at the position (I 1 , J f ) and each 
of the five pixel values Gobs(k, ig, jg) observed at points 
A to E. That is, for the given position (I', J') at which 
the true green light intensity Lg(I', J 1 ) is to be 

10 determined, the relationship between the true green light 
intensity Lg ( 1 1 , J 1 ) and each observed pixel value Gobs (k, 
i^/ jg) is represented by the spring model. 

Fig. 23 shows a method of estimating a true green light 
intensity Lgd 1 , J 1 ) at the position (I', J 1 ), based on the 

15 spring model associated with the position (I', J') of 

interest. In Fig. 23, an axis represents the green light 
intensity (G signal) . 

In the spring model shown in Fig. 23, one end of each 
of five springs is connected to an object V with a mass of 0, 

20 and the other end of each spring is connected to one of five 
pixel values Gobs(k, ig, jg) at points A to E. That is, 
five pixel values Gobs(k, ig, jg) at respective points A to 
E correspond to combinations of (k, ig, jg) that satisfy I' 
-l<x<I'+l and J 1 - 1 < y < J* +1 with respect to the 

25 position (x, y) in the reference coordinate system obtained 
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by converting, based on conversion parameters (0k, Tkx, Tky, 
Sk) , the position (ig - 0.5, jg - 0.5) for the given 
position (I 1 , J 1 ) shown in Fig. 22. 

The five springs have a natural length of 0 . A spring 
5 whose end is connected to a pixel value Gobs(k, ig, jg) 

observed at a point (x, y) has a spring constant equal to 
{V2 - F((x, y) , (I 1 , J ! ))}/(Mk x E) , as described earlier. 

The five springs pull the object V toward the five 
pixel values Gobs(k, ig, jg) , respectively, by forces 

10 proportional to the spring constants of the respective 

springs, and an equilibrium is achieved when the object V is 
at a particular location. The light intensity obtained at 
the location at which the equilibrium is achieved is 
employed as an estimate of the true green light intensity 

15 Lg(I', J'). The springs are in an equilibrium state when 
the sum of forces applied to the object V is equal to 0. 
That is, the equilibrium is achieved when the following 
equation is satisfied. 

_fV2-F((x.y). (I\ J')) 



x {Gobs(k, ig, jg) -Lg(I\ J')} 



= 0 



(Mk x E) 

20 • • • (25) 

Hereinafter, equation (25) will be referred to as a 
spring equation associated with the green light intensity. 
For the given position (I', J 1 ), the sum 2 in equation (25) 
is taken for all combinations (k, ig, jg) that satisfy I 1 - 
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1 < x < I' + 1 and J ! - 1 < y < J' +1 with respect to the 
position (x, y) in the reference coordinate system obtained 
by converting, based on conversion parameters (0k, Tkx, Tky, 
Sk) , the position (ig - 0.5, jg - 0.5). More specifically, 
5 in the example shown in Fig. 23, the sum is taken for five 
combinations of integers (k, ig, jg) corresponding to points 
A to E. 

Equation (25) is in the form of a linear equation 
including Lg(I', J') as an unknown. By solving equation 

10 (25), the true green light intensity Lg(I', J 1 ) at the 
position (I', J 1 ) can be determined. 

For the true red light intensity Lr ( I * , J') and the 
true blue light intensity Lb ( I 1 , J*) at the position (I 1 , 
J'), linear equations (26) and (27) can be written in a 

15 similar manner to that in which equation (25) is written for 
the pixel that senses the green light component as described 
above with reference to Figs. 18 to 23. 

-.fV2-F((x;y). (i\ J')) 



(Mk x E) 



x {Robs(k, ir, jr)-Lr(I\ J')) 



= 0 

• • • (26) 



20 Hereinafter, equation (26) will be referred to as a 

spring equation associated with the red light intensity. 
For the given position (I', J 1 ), the sum E in equation (26) 
is taken for all combinations (k, ir, jr) that satisfy I' - 
1 < x < I ' +1 and J' - 1 < y < J' +1 with respect to the 
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position (x, y) in the reference coordinate system obtained 
by converting, based on conversion parameters (Gk, Tkx, Tky, 
Sk) , the position (ir - 0.5, jr - 0.5). 

V2-F((x.y). (I\ J')) 



(Mk x E) 



x {Bobs(k, ib, jb)-Lb(I', J')} 



= 0 

• ■ ■ (27) 



Hereinafter, equation (27) will be referred to as a 
spring equation associated with the blue light intensity. 
For the given position (I', J'), the sum £ in equation (27) 
is taken for all combinations (k, ib, jb) that satisfy I ' - 

10 1 < x < I 1 + 1 and J' - 1 < y < J' +1 with respect to the 
position (x, y) in the reference coordinate system obtained 
by converting, based on conversion parameters Ok, Tkx, Tky, 
Sk) , the position (ib - 0.5, jb - 0.5) . 

Thus, in this third embodiment, for the given position 

15 (I 1 , J'), the true green light intensity Lg(I', J 1 ) , the 

true red light intensity Lr ( 1 1 , J 1 ), and the true blue light 
intensity Lb ( I ' , J 1 ) are determined based on pixel values 
corresponding to combinations (k, ib, jb) that satisfy I' - 
1 < x < I' +1 and J ! -l<y<J'+l with respect to the 

20 position (x, y) in the reference coordinate system obtained 
by converting, based on conversion parameters (9k, Tkx, Tky, 
Sk) , the position (i - 0.5, j - 0.5). 

Now, referring to a flow chart shown in Fig. 24, a 
correct image estimation process in step S4 shown in Fig. 2 
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according to a third embodiment of the present invention is 
described. 

First, in step S71, the calculation circuit 24 selects 
a position (I 1 , J') in the reference coordinate system 
5 (hereinafter, such a selected position will be referred to a 
position (I 1 , J') of interest). Note that the position (I 1 , 
J') of interest is at the center (i - 0.5, j - 0.5) of a (i- 
th, j-th) pixel of the first image taken as the reference 
image . 

10 Thereafter, the process proceeds from step S71 to step 

S72 . In step S72, the calculation circuit 24 determines all 
combinations of integers (k, ig, jg) that satisfy I 1 - 1 < x 
< I' + 1 and J' - 1 < y < J 1 +1 with respect to the 
position (x, y) in the reference coordinate system obtained 

15 by converting, based on conversion parameters (9k, Tkx, Tky, 
Sk) , the center position (ig - 0.5, jg - 0.5) of the pixel 
that senses a green component of the k-th image, for the 
position (I', J') of interest of each of first to N-th 
images. Thereafter, the process proceeds to step S73 . 

20 In step S73, the calculation circuit 24 produces a 

spring equation in the form (25) for green light intensity, 
using all combinations of (k, ig, jg) determined in step S72. 
Thereafter, the process proceeds to step S74 . 

In step S74, the calculation circuit 24 determines all 

25 combinations of integers (k, ir, jr) that satisfy I 1 - 1 < x 
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< 1 1 + 1 and J 1 -l<y<J ! +l with respect to the 
position (x, y) in the reference coordinate system obtained 
by converting, based on conversion parameters ' (0k, Tkx, Tky, 
Sk) , the center position (ir - 0.5, jr - 0.5) of the pixel 

5 that senses a red component of the k-th image, for the 
position (I 1 , J') of interest of each of first to N-th 
images. Thereafter, the process proceeds to step S75. 

In step S75, the calculation circuit 24 produces a 
spring equation in the form (26) for red light intensity, 
10 using all combinations of (k, ir, jr) determined in step S74 . 
Thereafter, the process proceeds to step S76 . 

In step S76, the calculation circuit 24 determines all 
combinations of integers (k, ib, jb) that satisfy I 1 - 1 < x 

< I" + 1 and J 1 - 1 < y < J' +1 with respect to the 

15 position (x, y) in the reference coordinate system obtained 
by converting, based on conversion parameters (0k, Tkx, Tky, 
Sk) , the center position (ib - 0.5, jb - 0.5) of the pixel 
that senses a blue component of the k-th image, for the 
position (I', J') of interest of each of first to N-th 

20 images. Thereafter, the process proceeds to step S77. 

In step S77, the calculation circuit 24 produces a 
spring equation in the form (27) for blue light intensity, 
using all combinations of (k, ib, jb) determined in step S76 . 
Thereafter, the process proceeds to step S78 . 

25 In step S78, the calculation circuit 24 solves the 
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spring equation (25) associated with the green light 
intensity determined in step S73, the spring equation (26) 
associated with the red light intensity determined in step 
S75, and the spring equation (27) associated with the blue 
5 light intensity determined in step S77, which are all in the 
form of a linear equation, thereby determining the true 
green light intensity Lg ( I 1 , J'), the true red light 
intensity Lr ( 1 1 , J 1 ), and the true blue light intensity 
Lb (I 1 , J') at the position (I', J') of interest. Thereafter, 

10 the process proceeds to step S79. 

In step S79, the calculation circuit 24 determines 
whether the above-described process is completed for all 
possible positions (I', J'), that is, whether the true green 
light intensity Lg(I', J 1 ), the true red light intensity 

15 Lr(I', J'), and the true blue light intensity Lb (I', J') 

have been determined by taking the center position of each 
of all pixels of the first image as the position (I 1 , J') of 
interest . 

If it is determined in step S79 that the process is not 
20 completed for all possible positions (I 1 , J') of interest, 
the process returns to step S71 to repeat steps S71 to S79. 
That is, the calculation circuit 24 selects, as a new 
position of interest, a position (I 1 , J') that has not yet 
been selected as the position of infest, and determines the 
25 true green light intensity Lg ( I ' , J 1 ), the true red light 
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intensity Lr(r, J'), and the true blue light intensity 
Lb (I', J') at the position (I 1 , J ! ) of interest. 

On the other hand in the case in which it is determined 
in step S79 that the process is completed by taking all 
possible position (I 1 , J') as the position of interest, the 
process proceeds to step S80. In step S80, the calculation 
circuit 24 estimates a correct image (signal) from the true 
green light intensity Lg(I', J'), the true red light 
intensity Lr ( 1 1 , J ! ), and the true blue light intensity 
Lb (I', J') determined in step S78 . The resultant image is 
supplied as an output image to the D/A converter 9 or the 
codec 12, and the current process is exited. For example, 
for the (i-th, j-th) pixel, the calculation circuit 24 
estimates (produces) the correct image signal thereof from 
the true green light intensity Lg(i - 0.5, j - 0.5) 
determined as the green value (G signal) in step S78, the 
true red light intensity Lr(i - 0.5, j - 0.5) determined as 
the red value (R signal) in step S78, and the true blue 
light intensity Lb(i - 0.5, j - 0.5) determined as the blue 
value (B signal) in step S78 . The calculation circuit 24 
estimates the output image by estimating the image signal 
for all pixels whose center is located at the selected 
position ( I ' , J 1 ) . 

In the third embodiment, as described above, data 
sensed by each pixel of the imaging device 4 is regarded as 
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point -sampled data, the relationship between a pixel value 
observed at the center of each pixel and an ideal image 
signal of a sharp image including no blur due to a camera 
shake is represented by the spring model, and a sharp image 
5 likely to be a true image is determined based on the spring 
model . 

Now, a correct image estimation process according to a 
fourth embodiment of the present invention is described. In 
this fourth embodiment, the third embodiment described above 

10 with reference to Fig. 24 is partially improved. 

That is, in the third embodiment shown in Fig. 24, in 
step S72, the calculation circuit 24 determines all 
combinations of integers (k, ig, jg) that satisfy I* - 1 < x 
< I ' + 1 and J 1 -l<y<J'+l with respect to the 

15 position (x, y) in the reference coordinate system obtained 
by converting, based on conversion parameters (9k, Tkx, Tky, 
Sk) , the center position (ig - 0.5, jg - 0.5) of the pixel 
that senses a green component of the k-th image, for the 
position (I', J') of interest of each of first to N-th 

20 images. In step S73, the calculation circuit 24 produces a 
spring equation in the form (25) for green light intensity, 
using all combinations of integers (k, ig, jg) determined in 
step S72 . Similarly, for the pixels that sense the red or 
blue component, combinations of integers (k, ig, jg) are 

25 determined for the position (I', J') of interest, and the 
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spring equation in the form of (26) or (27) is produced 
using all determined combinations of integers (k, ig, jg) . 

There is a possibility that data of a particular pixel 
of a particular image of the first to N-th images is 
5 unreliable depending on the state in which the images are 

captured. In the digital camera 1 shown in Fig. 1, N images 
are taken and stored in the frame memory 22. Therefore, 
even if data with low reliability is discarded, the amount 
of remaining data can be sufficiently large. 

10 In view of the above, in a fourth embodiment, only data 

with high reliability is used (and data with low reliability 
is discarded) to produce an image with better sharpness. 
More specifically, in the fourth embodiment, as many 
combinations of integers (k, ig, jg) as a predetermined 

15 number L are extracted as data with high reliability from 
all (k, ig, jg) determined for a position (I', J') of 
interest . The number L may be fixed to a predetermined 
value such as 8 or may be set to a variable value specified 
by a user. The calculation circuit 24 calculates the mean 

20 value of pixel values Gobs(k, ig, jg) of all combinations of 
integers (k, ig, jg) and selects L (= 8) combinations of 
integers (k, ig, jg) whose pixel value Gobs (k, ig, jg) is 
closest to the mean value. The selected L combinations are 
used to determine the true green light intensity Lg(I', J') 

25 at the position (I 1 , J 1 ). 
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Thus, in this fourth embodiment, the spring equations 
(25) , (26) , and (27) associated with the green, red and blue 
light intensities used in the third embodiment are 
respectively rewritten into equations (28) , (29) and (30) , 
as shown below. 

72-F((x,y), (I\ J')) 



(Mk x E) 



x {Gobs(k, ig, jg)-Lg(I', J')} 



= 0 



(28) 



where £ represents the sum taken for L combinations of 
integers (k, ig, jg) whose pixel value Gobs(k, ig, jg) is 
closest to the mean value of pixel values Gobs(k, ig, jg) of 
all combinations of integers (k, ig, jg) determined for the 
position (I', J') of interest. 

72"-F((x,y), <!', J')) 



(Mk x E) 



x {Robs(k, ir, jr)-Lr(I\ J')} 



= 0 



(29) 



where £ represents the sum taken for L combinations of 
integers (k, ir, jr) whose pixel value Robs (k, ir, jr) is 
closest to the mean value of pixel values Robs(k, ir, jr) of 
all combinations of integers (k, ir, jr) determined for the 
position (I', J') of interest. 

V2-F((x.y). (I\ J')) 



(Mk x E) 



x {Bobs(k, ib, jb) -Lb(I\ J')} 



= 0 



(30) 



where S represents the sum taken for L combinations of 
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integers (k, ib, jb) whose pixel value Bobs(k, ib, jb) is 
closest to the mean value of pixel values Bobs (k, ib, jb) of 
all combinations of integers (k, ib, jb) determined for the 
position (I 1 , J 1 ) of interest. 
5 Now, referring to a flow chart shown in Fig. 25, a 

correct image estimation process in step S4 shown in Fig. 2 
according to a fourth embodiment of the present invention is 
described . 

First, in step S91, the calculation circuit 24 selects 
10 a position (I', J') in the reference coordinate system 

(hereinafter, such a selected position will be referred to a 
position (I 1 , J 1 ) of interest). Note that the position (I', 
J') of interest is at the center (i - 0.5, j - 0.5) of a (i- 
th, j-th) pixel of the first image taken as the reference 
15 image. 

Thereafter, the process proceeds from step S91 to step 
S92 . In step S92, the calculation circuit 24 determines all 
combinations of integers (k, ig, jg) that satisfy I 1 - 1 < x 
< I ' + 1 and J 1 -l<y<J ! +l with respect to the 

20 position (x, y) in the reference coordinate system obtained 
by converting, based on conversion parameters (0k, Tkx, Tky, 
Sk) , the center position (ig - 0.5, jg - 0.5) of the pixel 
that senses a green component of the k-th image, for the 
position (I', J ! ) of interest of each of first to N-th 

25 images. Thereafter, the process proceeds to step S93 . 
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In step S93, the calculation circuit 24 calculates the 
mean value of pixel values Gobs(k, ig, jg) for all 
combinations of integers (k, ig, jg) determined in step S92 # 
and selects L combinations of integers (k, ig, jg) whose 
5 pixel value Gobs(k, ig, jg) is closest to the mean value. 
Thereafter, the process proceeds to step S94 . That is, in 
step S93 described above, pixel values Gobs(k, ig, jg) that 
are not close to the mean value taken for all combinations 
of integers (k, ig, jg) determined in step S92 are regarded 

10 as data with low reliability and they are discarded. In a 
case in which the total number of combinations of integers 
(k, ig, jg) determined in step S92 is less than L, all 
combinations of integers (k, ig, jg) are selected (without 
discarding any data) . 

15 In step S94, the calculation circuit 24 produces a 

spring equation in the form (28) for green light intensity, 
using L combinations of (k, ig, jg) selected in step S93 . 
Thereafter, the process proceeds to step S95 . 

In step S95, the calculation circuit 24 determines all 

20 combinations of integers (k, ir, jr) that satisfy I' - 1 < x 
< I' + 1 and J 1 - 1 < y < J 1 +1 with respect to the 
position (x, y) in the reference coordinate system obtained 
by converting, based on conversion parameters (0k, Tkx, Tky, 
Sk) , the center position (ir - 0.5, jr - 0.5) of the pixel 

25 that senses a red component of the k-th image, for the 
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position (I 1 , J') of interest of each of first to N-th 
images. Thereafter, the process proceeds to step S96 . 

In step S96, the calculation circuit 24 calculates the 
mean value of pixel values Robs(k, ir, jr) for all 
5 combinations of integers (k, ir, jr) determined in step S96, 
and selects L combinations of integers (k, ir, jr) whose 
pixel value Robs(k, ir, jr) is closest to the mean value. 
Thereafter, the process proceeds to step S97 . That is, in 
step 396 described above, pixel values Robs(k, ir, jr) that 

10 are not close to the mean value taken for all combinations 
of integers (k, ir, jr) determined in step S95 are regarded 
as data with low reliability and they are discarded. In a 
case in which the total number of combinations of integers 
(k, ir, jr) determined in step S95 is less than L, all 

15 combinations of integers (k, ir, jr) are selected (without 
discarding any data) . 

In step S97, the calculation circuit 24 produces a 
spring equation in the form (29) for red light intensity, 
using L combinations of (k, ir, jr) selected in step S96 . 

20 Thereafter, the process proceeds to step S98 . 

In step S98, the calculation circuit 24 determines all 
combinations of integers (k, ib, jb) that satisfy I' - 1 < x 
< I 1 + 1 and J' - 1 < y < J' +1 with respect to the 
position (x, y) in the reference coordinate system obtained 

25 by converting, based on conversion parameters (0k, Tkx, Tky, 
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Sk) , the center position (ib - 0.5, jb - 0.5) of the pixel 
that senses a blue component of the k-th image, for the 
position (I', J') of interest of each of first to N-th 
images. Thereafter, the process proceeds to step S99. 
5 In step S99, the calculation circuit 24 calculates the 

mean value of pixel values Bobs (k, ib, jb) for all 
combinations of integers (k, ib, jb) determined in step S98, 
and selects L combinations of integers (k, ib, jb) whose 
pixel value Bobs(k, ib, jb) is closest to the mean value. 

10 Thereafter, the process proceeds to step S100 . That is, in 
step S99 described above, pixel values Bobs(k, ib, jb) that 
are not close to the mean value taken for all combinations 
of integers (k, ib, jb) determined in step S98 are regarded 
as data with low reliability and they are discarded. In a 

15 case in which the total number of combinations of integers 
(k, ib, jb) determined in step S98 is less than L, all 
combinations of integers (k, ib, jb) are selected (without 
discarding any data) . 

In step S100, the calculation circuit 24 produces a 

20 spring equation in the form (30) for blue light intensity, 
using L combinations of (k, ib, jb) selected in step S99. 
Thereafter, the process proceeds to step S101. 

In step S101, the calculation circuit 24 solves the 
spring equation (28) associated with the green light 

25 intensity determined in step S94, the spring equation (29) 
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associated with the red light intensity determined in step 
S97 , and the spring equation (30) associated with the blue 
light intensity determined in step S100, which are all in 
the form of a linear equation, thereby determining the true 
5 green light intensity Lg(I', J ! ), the true red light 

intensity Lr(I', J 1 ), and the true blue light intensity 

Lb(I' , J 1 ) at the position (I', J') of interest. Thereafter, 

the process proceeds to step S102 . 

In step S102, the calculation circuit 24 determines 

10 whether the above -described process is completed for all 

possible positions (I 1 , J 1 ), that is, whether the true green 
light intensity Lg ( I ' , J 1 ), the true red light intensity 
Lr(I' , J')/ and the true blue light intensity Lb (I 1 , J 1 ) 
have been determined by taking the center position of each 

15 of all pixels of the first image as the position (I 1 , J 1 ) of 
interest . 

If it is determined in step S102 that the process is 
not completed for all possible positions (I 1 , J') of 
interest, the process returns to step S91 to repeat steps 

20 S91 to S102. That is, the calculation circuit 24 selects, 
as a new position of interest, a position (I 1 , J') that has 
not yet been selected as the position of infest, and 
determines the true green light intensity Lg(I', J'), the 
true red light intensity Lr(I', J'), and the true blue light 

25 intensity Lb(I', J') at the position (I 1 , J') of interest. 
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On the other hand in the case in which it is determined 
in step S102 that the process is completed by taking all 
possible position (I 1 , J') as the position of interest, the 
process proceeds to step S103 . In step S103, the 
5 calculation circuit 24 estimates a correct image (signal) 

from the true green light intensity Lg ( I ' , J'), the true red 
light intensity Lr(I', J 1 ), and the true blue light 
intensity Lb(I', J ! ) determined in step S101. The resultant 
image is supplied as an output image to the D/A converter 9 

10 or the codec 12, and the current process is exited. For 

example, for the (i-th, j-th) pixel, the calculation circuit 
24 estimates a correct image signal thereof from the true 
green light intensity Lg(i - 0.5, j - 0.5) determined as the 
green value (G signal) in step S101, the true red light 

15 intensity Lr(i - 0.5, j - 0.5) determined as the red value 
(R signal) in step S101, and the true blue light intensity 
Lb(i - 0.5, j - 0.5) determined as the blue value (B signal) 
in step S101. The calculation circuit 24 estimates a 
correct image by performing the above -described process for 

20 all pixels whose center is located at the selected position 
(I 1 , J ! ) . 

In the fourth embodiment, as described above, pixel 
values close to the mean value are employed as reliable data, 
and the spring model is employed to only the employed 
25 reliable data. This makes it possible to obtain a sharper 
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image more likely to be a correct image than can be obtained 
by the third embodiment described above. 

Now, a correct image estimation process according to a 
fifth embodiment of the present invention is described. In 
5 this fifth embodiment, as in the fourth embodiment, as many 
combinations of integers (k, ig, jg) as a predetermined 
number L are extracted as reliable data from all 
combinations of integers (k, ig, jg) determined for a 
position (I', J 1 ) of interest. The true green light 

10 intensity Lg(I f , J'), the true red light intensity Lr(T , 
J 1 ), and the true blue light intensity Lb(I', J ! ) are 
determined by solving spring equations associated with green, 
red, and blue light intensities each using L reliable data. 
In the fourth embodiment described above, for example, 

15 in the determination of the true green light intensity Lg ( I 1 , 
J'), the mean value of pixel values Gobs(k, ig, jg) of all 
combinations of integers (k, ig, jg) is determined by the 
calculation circuit 24, and L combinations of integers (k, 
ig/ jg) whose pixel value Gobs(k, ig, jg) is closest to the 

20 mean value are selected. 

On the other hand, in the fifth embodiment, for a given 
position (I 1 , J') of interest, the calculation circuit 24 
extracts reliable data by selecting L combinations of 
integers (k, ig, jg) that are shortest in distance from the 

25 position of interest (I 1 , J') to the position (x, y) in the 
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reference coordinate system obtained by converting, based on 
conversion parameters (0k, Tkx, Tky, Sk) , the center 
position (ig - 0.5, jg - 0.5) of the pixel which senses the 
green component of the k-th image. This is because, as 
5 described earlier with reference to Fig. 19, the 

approximation error that can occur when the true green light 
intensity Lg(I ! , J') at the position (I', J') of interest is 
approximated by the pixel value Gobs(k, ig, jg) observed at 
the position (x, y) increases with the distance between the 

10 position (I', J') of interest and the position (x, y) . In 
other words, pixel values Gobs(k, ig, jg) observed at 
positions (x, y) distant from the position (I 1 , J') of 
interest are low in reliability. To determined the distance 
between two points (xl, yl) and (x2, y2) , for example, the 

15 function F((xl, yl) , (x2, y2)) described earlier with 
reference to Fig. 19 can be used. 

In the fifth embodiment, spring equations (31), (32), 
and (33) associated with green, red, and blue light 
intensities shown below are used instead of the spring 

20 equations (25), (26), and (27) used in the third embodiment. 

UV2-F((x.y). (I'. J')) 



x {Gobs(k, ig, jg) -Lgd', J')} 



= 0 



(Mk x E) 

• • • (31) 

where 2 denotes the sum taken for L combinations of integers 
(k, ig, jg) that are shortest in distance from the position 
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10 



15 



20 



(I 1 , J ! ) of interest to the position (x, y) in the reference 
coordinate system obtained by converting, based on 
conversion parameters (6k, Tkx, Tky, Sk) , the center 
position (ig - 0.5, jg - 0.5) of the pixel which senses the 
green component of the k-th image. 

_fV2-F((x f y). (I\ J')) 



(Mk x E) 



x {Robs(k, ir. jr)-Lr(I\ J')} 



= 0 



(32) 



where S denotes the sum taken for L combinations of integers 
(k, ir, jr) that are shortest in distance from the position 
(I 1 , J') of interest to the position (x, y) in the reference 
coordinate system obtained by converting, based on 
conversion parameters Ok, Tkx, Tky, Sk) , the center 
position (ir - 0.5, jr - 0.5) of the pixel which senses the 
red component of the k-th image. 

^fV2-F((x.y). U\ J')) 



(Mk x E) 



x {Bobs(k, ib, jb)-Lb(I\ J')} 



= 0 



(33) 



where S denotes the sum taken for L combinations of integers 
(k, ib, jb) that are shortest in distance from the position 
(I', J') of interest to the position (x, y) in the reference 
coordinate system obtained by converting, based on 
conversion parameters (8k, Tkx, Tky, Sk) , the center 
position (ib - 0.5, jb - 0.5) of the pixel which senses the 
blue component of the k-th image. 
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Now, referring to a flow chart shown in Fig. 26, a 
correct image estimation process in step S4 shown in Fig. 2 
according to a fifth embodiment of the present invention is 
described. 

5 First, in step S121, the calculation circuit 24 selects 

a position (I 1 , J') in the reference coordinate system 
(hereinafter, such a selected position will be referred to a 
position (I 1 , J 1 ) of interest). Note that the position (I 1 , 
J*) of interest is at the center (i - 0.5, j - 0.5) of a (i- 
10 th, j-th) pixel of the first image taken as the reference 
image . 

Thereafter, the process proceeds from step S121 to step 
S122 . In step S122, the calculation circuit 24 determines 
all combinations of integers (k, ig, jg) that satisfy I 1 - 1 

15 < x < I 1 + 1 and J ' -l<y<J'+l with respect to the 

position (x, y) in the reference coordinate system obtained 
by converting, based on conversion parameters (Gk, Tkx, Tky, 
Sk) , the center position (ig - 0.5, jg - 0.5) of the pixel 
that senses a green component of the k-th image, for the 

20 position (I', J') of interest of each of first to N-th 
images. Thereafter, the process proceeds to step S123 . 

In step S123, from all combinations of integers (k, ig, 
jg) determined in step S122, the calculation circuit 24 
selects L combinations of integers (k, ig, jg) that are 

25 shortest in distance from the position of interest (I 1 , J 1 ) 
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to the position (x, y) in the reference coordinate system 
obtained by converting, based on conversion parameters (0k, 
Tkx, Tky, Sk) , the center position (ig - 0.5, jg - 0.5) of 
the pixel which senses the green component of the k-th image. 
5 Thereafter, the process proceeds to step S124 . That is, in 
step S123 described above, of all combinations of integers 
(k, ig, jg) determined in step S122, those that are long in 
distance between the position (x, y) and the position (I 1 , 
J 1 ) of interest are regarded as data with low reliability 

10 and they are discarded. In a case in which the total number 
of combinations of integers (k, ig, jg) determined in step 
S122 is less than L, all combinations of integers (k, ig, 
jg) are selected (without discarding any data) . 

In step S124, the calculation circuit 24 produces a 

15 spring equation in the form (31) for green light intensity, 
using L combinations of (k, ig, jg) selected in step S123. 
Thereafter, the process proceeds to step S125. 

In step S125, the calculation circuit 24 determines all 
combinations of integers (k, ir, jr) that satisfy I 1 - 1 < x 

20 < I 1 + 1 and J' - 1 < y < J • +1 with respect to the 

position (x, y) in the reference coordinate system obtained 
by converting, based on conversion parameters (0k, Tkx, Tky, 
Sk) , the center position (ir - 0.5, jr - 0.5) of the pixel 
that senses a red component of the k-th image, for the 

25 position (I 1 , J 1 ) of interest of each of first to N-th 
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images. Thereafter, the process proceeds to step S126. 

In step S126, from all combinations of integers (k, ir, 
jr) determined in step S125, the calculation circuit 24 
selects L combinations of integers (k, ir, jr) that are 
5 shortest in distance from the position of interest (I 1 , J ! ) 
to the position (x, y) in the reference coordinate system 
obtained by converting, based on conversion parameters (0k, 
Tkx, Tky, Sk) , the center position (ir - 0.5, jr - 0.5) of 
the pixel which senses the green component of the k-th image. 

10 Thereafter, the process proceeds to step S127. That is, in 
step S126 described above, of all combinations of integers 
(k, ir, jr) determined in step S125, those that are long in 
distance between the position (x, y) and the position (I', 
J') of interest are regarded as data with low reliability 

15 and they are discarded. In a case in which the total number 
of combinations of integers (k, ir, jr) determined in step 
S125 is less than L, all combinations of integers (k, ir, 
jr) are selected (without discarding any data) . 

In step S127, the calculation circuit 24 produces a 

20 spring equation in the form (32) for red light intensity, 
using L combinations of (k, ir, jr) selected in step S126. 
Thereafter, the process proceeds to step S128. 

In step S128, the calculation circuit 24 determines all 
combinations of integers (k, ib, jb) that satisfy I' - 1 < x 

25 < I 1 + 1 and J 1 - 1 < y < J' +1 with respect to the 
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position (x, y) in the reference coordinate system obtained 
by converting, based on conversion parameters (0k, Tkx, Tky, 
Sk) , the center position (ib - 0.5, jb - 0.5) of the pixel 
that senses a blue component of the k-th image, for the 
5 position (I 1 , J') of interest of each of first to N-th 
images. Thereafter, the process proceeds to step S129. 

In step S129, from all combinations of integers (k, ib, 
jb) determined in step S128, the calculation circuit 24 
selects L combinations of integers (k, ib, jb) that are 

10 shortest in distance from the position of interest (I*, J') 
to the position (x, y) in the reference coordinate system 
obtained by converting, based on conversion parameters (9k, 
Tkx, Tky, Sk) , the center position (ib - 0.5, jb - 0.5) of 
the pixel which senses the green component of the k-th image. 

15 Thereafter, the process proceeds to step S130. That is, in 
step S129 described above, of all combinations of integers 
(k, ib, jb) determined in step S128, those that are long in 
distance between the position (x, y) and the position (I', 
J') of interest are regarded as data with low reliability 

20 and they are discarded. In a case in which the total number 
of combinations of integers (k, ib, jb) determined in step 
S128 is less than L, all combinations of integers (k, ib, 
jb) are selected (without discarding any data) . 

In step S130, the calculation circuit 24 produces a 

25 spring equation in the form (33) for blue light intensity, 
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using L combinations of (k, ib, jb) selected in step S129. 
Thereafter, the process proceeds to step S131. 

In step S131, the calculation circuit 24 solves the 
spring equation (31) associated with the green light 
5 intensity determined in step S124, the spring equation (32) 
associated with the red light intensity determined in step 
S127, and the spring equation (33) associated with the blue 
light intensity determined in step S13 0, which are all in 
the form of a linear equation, thereby determining the true 

10 green light intensity Lg(I ! , J 1 ), the true red light 

intensity Lr ( I ' , J'), and the true blue light intensity 

Lb (I 1 , J 1 ) at the position (I 1 , J 1 ) of interest. Thereafter, 

the process proceeds to step S132 . 

In step S132, the calculation circuit 24 determines 

15 whether the above-described process is completed for all 

possible positions (I', J 1 ), that is, whether the true green 
light intensity Lg(I ! , J 1 ), the true red light intensity 
Lr(I*, J 1 ), and the true blue light intensity Lb ( I ■ , J') 
have been determined by taking the center position of each 

20 of all pixels of the first image as the position (I', J') of 
interest . 

If it is determined in step S132 that the process is 
not completed for all possible positions (I', J ! ) of 
interest, the process returns to step S121 to repeat steps 
25 S121 to S132. That is, the calculation circuit 24 selects, 
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as a new position of interest, a position (I 1 , J') that has 
not yet been selected as the position of infest, and 
determines the true green light intensity Lg(I', J'), the 
true red light intensity Lr(I', J'), and the true blue light 
5 intensity Lb (I 1 , J') at the position (I 1 , J 1 ) of interest. 

On the other hand in the case in which it is determined 
in step S132 that the process is completed by taking all 
possible position (I 1 , J') as the position of interest, the 
process proceeds to step S133 . In step S133, the 

10 calculation circuit 24 estimates a correct image (signal) 

from the true green light intensity Lg(I', J'), the true red 
light intensity Lr(r , J ! ), and the true blue light 
intensity Lb ( I ' , J') determined in step S131. The resultant 
image is supplied as an output image to the D/A converter 9 

15 or the codec 12, and the current process is exited. For 

example, for the (i-th, j-th) pixel, the calculation circuit 
24 estimates the correct image signal thereof from the true 
green light intensity Lg(i - 0.5, j - 0.5) determined as the 
green value (G signal) in step S131, the true red light 

20 intensity Lr(i - 0.5, j - 0.5) determined as the red value 
(R signal) in step S131, and the true blue light intensity 
Lb(i - 0.5, j - 0.5) determined as the blue value (B signal) 
in step S131. The calculation circuit 24 estimates the 
output image by estimating the image signal for all pixels 

25 whose center is located at the selected position (I', J 1 ). 
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In the fifth embodiment, as described above, pixel 
values observed at positions close to the position (I', J 1 ) 
of interest are selected as reliable data, and the spring 
model is applied only to the selected data. This makes it 
5 possible to obtain a more ideal image with better sharpness 

than can be obtained by the third embodiment described above . 

Now, a correct image estimation process according to a 
sixth embodiment of the present invention is described. 
In this sixth embodiment, the third embodiment 
10 described above is further improved. That is, in this sixth 
embodiment, edge parts of an image are detected, and spring 
equations in improved forms based on spring equations (25) , 
(26) , and (27) associated with green, red and blue light 
intensities are applied to pixel values in the detected edge 
15 parts. 

Of green, red, and blue components, a reference 
coordinate system associated with the green component (G 
signal) is shown in Fig. 27. 

In the example, shown in Fig. 27, pixel values with 

2 0 bright green are observed in an area to the right of a 

boundary (edge) 51, and pixel values with dark green are 
observed in an area to the left of the boundary (edge) 51. 
Each pixel value of green is represented in, for example, 8 
bits. The bright green pixel value is equal to, for example, 

2 5 24 0, and the dark green pixel value is equal to, for example, 
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16 . 

In Fig. 27, open circles 52-1 to 52-10, 53-1 to 53-9, 
and 54 indicate center positions (ig - 0.5, jg - 0.5) of 
(ig-th, jg-th) pixels of the k-th image converted, based on 
5 conversion parameters (6k, Tkx, Tky, Sk) , to representation 
in the reference coordinate system. At those positions, 
green pixel values Gobs(k, ig, jg) are observed. Note that 
(k, ig, jg) is different among open circles 52-1 to 52-10, 
53-1 to 53-9, and 54. 
10 Each solid circle at an intersection of a lattice 

represented by dotted lines in Fig. 27 indicates a position 
(I 1 , J') of green light intensity Lg ( I ' , J') to be 
determined by the calculation circuit 24 . As described 
earlier, the position (I 1 , J') denotes the center position 
15 (i - 0.5, j - 0.5) of the (i-th, j-th) pixel of the first 

image employed as the reference image. The distance between 
adjacent Lg(I f , J 1 ) is equal to 1 in both X and Y directions, 
and thus the pixel size is equal to 1 in both X and Y 
directions . 

20 At positions denoted by open circles 52-1 to 52-8 in a 

region 61 shown in Fig. 27, bright green pixel values Gobs(k, 
j9) equal to about 240 are observed. 

At positions denoted by open circles 53-1 to 53-7 in a 
region 62 shown in Fig. 27, dark green pixel values Gobs(k, 
25 ig, jg) equal to about 16 are observed. 
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In Fig. 27, the position indicated by the open circle 
54 is on the boundary 51. At such a position, because of 
the characteristic of the on-chip lens, the pixel receives 
equally both bright green light with a level of 24 0 and dark 
5 green light with a level of 16, and thus the pixel value 

Gobs(k, ig, jg) observed at the position denoted by the open 
circle 54 is equal to the mean value of 240 and 16, that is, 
128 (= (240 + 16) /2) . 

The pixel values Gobs(k, ig, jg) observed at positions 

10 denoted by open circles 52-1 to 52-8, 53-1 to 53-7 and 54 
generally include an error component, and thus, from a 
strict point of view, pixel values are, for example, about 
240, about 16, about 128, and so on. However, for the 
purpose of simplicity, error components are ignored and 

15 pixel values are simply expressed as 240, 16, 128, and so on. 
For example, when the position denoted by the solid 
circle 71 in Fig. 27 is selected as the position (I', J') of 
interest, if the third embodiment described above is used, 
the true green light intensity Lg(I', J') at the position 

20 (I', J') of interest denoted by the solid circle 71 is 
determined as follows. 

That is, when the position denoted by the solid circle 
71 is selected as the position (I 1 , J 1 ) of interest, pixel 
values Gobs(k, ig, jg) observed in the vicinity of the 

25 position (I 1 , J 1 ) of interest, such as those observed at 
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positions denoted by open circles 52-3, 52-6, 52-10, and 54, 
are employed as Gobs (k, ig, jg) in equation (25) . In this 
case, the green light intensity Lg ( I ' , J') obtained by 
solving equation (25) becomes lower than 240 because the 
5 true green light intensity Lg(I ! , J') is pulled (affected) 
by the pixel value of 128 observed at the position denoted 
by the open circle 54. However, because the position 
denoted by the solid circle 71 is located in an area to the 
right of the boundary 51 where the green pixel value 

10 observed is as high as 240, the really true green light 

intensity Lg(I f , J') is 240. Thus, it is desirable that the 
green light intensity Lg(I', J') obtained by solving 
equation (25) be equal to 240. 

When the position denoted by the solid circle 72 is 

15 selected as the position (I', J 1 ) of interest, pixel values 

Gobs(k, ig, jg) observed in the vicinity of the position (I', 
J 1 ) of interest, such as those observed at positions denoted 
by open circles 53-2, 53-3, 53-5, 53-9 and 54 are employed 
as Gobs(k, ig, jg) in equation (25). In this case, the true 

20 green light intensity Lg(I', J') obtained by solving 

equation (25) becomes greater than 16 because the true green 
light intensity Lg ( I 1 , J') is pulled (affected) by the pixel 
value of 128 observed at the position denoted by the open 
circle 54. However, because the position denoted by the 

25 solid circle 72 is located in an area to the left of the 
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boundary 51 where the green pixel value observed is as low 
as 16, the really true green light intensity Lg(I', J') is 
16. Thus, it is desirable that the green light intensity 
Lg(I', J') obtained by solving equation (25) be equal to 16. 
5 The above discussion can also be applied to locations 

denoted by solid circles 73 and 74. As discussed above, in 
images produced by the correct image estimation process 
according to the third embodiment, in edge parts such as 
that in which the boundary 51 shown in Fig. 27 exists, there 
10 is a possibility that an edge is blurred (that is, 

differences among pixel values of adjacent pixels become 
small) . 

In the sixth embodiment, to avoid the above problem, an 
edge part including the boundary 51 shown in Fig. 27 is 
15 detected, and the special process is performed on pixel 

values Gobs(k, ig, jg) in the detected edge part, that is, 
pixel values Gobs(k, ig, jg) substituted into the spring 
equation (25) associated with the green light intensity are 
changed . 

20 The method according to the sixth embodiment is 

described in further detail below for the case in which the 
green light intensity (G signal) is determined with 
reference to Fig. 27. 

First, the calculation circuit 24 detects an edge part 

25 based on all pixel values Gobs(k, ig, jg) at positions 
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converted into the reference coordinate system. That is, 
the calculation circuit 24 determines whether there is an 
edge extending in a vertical direction, in a horizontal 
direction, in a direction from upper left to lower right, or 
5 in a direction from lower right to lower left. 

Referring to Fig. 28, a method of determining whether 
there is a vertical edge is described below. 

Fig. 2 8 shows a reference coordinate system. In Fig. 
28, open circles 81-1 to 81-10, 82-1 to 82-9, and 83 

10 indicate center positions (ig - 0.5, jg - 0.5) of (ig-th, 
jg-th) pixels of the k-th image converted, based on 
conversion parameters (0k, Tkx, Tky, Sk) , to representation 
in the reference coordinate system, and pixel values Gobs(k, 
i<?/ j9) are observed at those positions. Note that (k, ig, 

15 jg) is different among open circles 81-1 to 81-10, 82-1 to 
82-9, and 83 . 

Each solid circle at an intersection of a lattice 
represented by dotted lines in Fig. 28 indicates a position 
(I 1 , J') of green light intensity Lg ( I ' , J 1 ) to be 

20 determined by the calculation circuit 24. As described 

earlier, the position (I 1 , J 1 ) denotes the center position 
(i - 0.5, j - 0.5) of the (i-th, j-th) pixel of the first 
image employed as the reference image. As in the example 
shown in Fig. 27, the distance between adjacent Lg(I', J') 

25 is equal to 1 in both X and Y directions, and thus the pixel 
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size is equal to 1 in both X and Y directions. 

For example, the calculation circuit 24 selects a 
position denoted by an open circle 83 shown in Fig. 28 as a 
position of interest, and determines whether there is a 
5 vertical edge such as an edge 94 shown in Fig. 28. Herein, 
let us assume that the position (x, y) denoted by the open 
circle 83 satisfy inequality I-0.5<x<I+0.5 and J - 
0.5<y<J+0.5. In the example shown in Fig. 28, the 
position (x, y) denoted by the open circle 83 is located in 

10 a rectangular area All whose four corners are located at (I 
+ 0.5, J - 0.5) denoted by a solid circle 84, (I - 0.5, J - 
0.5) denoted by a solid circle 85, (I + 0.5, J + 0..5) 
denoted by a solid circle 86, and (I - 0.5, J + 0.5) denoted 
by a solid circle 87. I and J are integers as in the 

15 previous embodiments. 

The calculation circuit 24 defines nine regions each 
having a size of 1 equal to the size of one pixel such that 
a region All including an open circle 83 of interest is 
located at the center of these nine regions. These nine 

20 regions are: a region A00 defined by I - 1.5 < x < I - 0.5 

and J-i.5<y<J-0.5;a region A01 defined by I - 1.5 < 
x < I - 0.5 and J-0.5<y<J+0.5;a region A02 defined 
by I - 1.5 < x < I - 0.5 and J+0.5<y<J+1.5; a region 
A10 defined by I - 0.5 < x < I + 0.5 and J-1.5<y<J- 

25 0.5; the region All defined by I - 0.5 < x < I + 0.5 and J - 
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0.5<y<J+0.5; a region A12 defined by I - 0.5 < x < I + 
0.5 and J+0.5<y<J+1.5; a region A20 defined by I + 
0.5<x<I+0.5 and J-1.5<y<J-0.5;a region A21 
defined by I + 0.5 < x < I + 1.5 and J-0.5<y<J+0.5; 
5 and a region A22 defined by I+0.5<x<I+0.5 and J + 

0.5 < y < J +1.5. The current position of interest denoted 
by the open circle 83 is in the region All. 

The total area including the nine regions A00 to A22 
defined for the position of interest denoted by the open 

10 circle 83 is referred to as an edge judgment region. 

To determine whether there is a horizontal edge such as 
the edge 94, the calculation circuit 24 calculates the mean 
value and the variance of pixel values Gobs(k, ig, jg) at 
locations denoted by open circles in a left region 91 

15 including a region A00, a region A01, and a region A02 in 
the edge judgment region (hereinafter, this region 91 will 
be referred to simply as the left region 91) , the mean value 
and the variance of pixel values Gobs(k, ig, jg) at 
locations denoted by open circles in a central region 92 

20 including a region A10, a region All, and a region A12 in 
the edge judgment region (hereinafter, this region 92 will 
be referred to simply as the central region 92) , and the 
mean value and the variance of pixel values Gobs(k, ig, jg) 
at locations denoted by open circles in a right region 93 

25 including a region A20, a region A21, and a region A22 in 
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the edge judgment region (hereinafter, this region 93 will 
be referred to simply as the right region 93). 

That is, the calculation circuit 24 determines the mean 
value EGO of pixel values Gobs(k, ig, jg) at positions 
5 denoted by open circles 82-1 to 82-7 in a left-hand region 
91 and also determines the variance SGO of those pixel 
values Gobs(k, ig, jg) . The calculation circuit 24 also 
determines the mean value EGO 1 1 of pixel values Gobs(k, ig, 
jg) at positions denoted by open circles 81-9, 81-10, 82-8, 

10 82-9, and 83 in a central region 92. Furthermore, the 

calculation circuit 24 determines the mean value EGO ' of 
pixel values Gobs(k, ig, jg) at positions denoted by open 
circles 81-1 to 81-8 in a right-hand region 93 and also 
determines the variance SGO' of those pixel values Gobs(k, 

15 ig, jg) . 

By way of example, if there is a vertical edge 94 close 
to the position of interest denoted by the open circle 83, 
the following inequality (a) holds for the mean value EGO of 
the left region 91, the mean value EGO 1 ' of the central 
20 region 92, and the mean value EGO' of the right region 93, 

and thus the calculation circuit 24 determines that there is 
a vertical edge when inequality (a) is satisfied. 

EGO < EGO" < EGO' or EGO' < EGO" < EGO (a) 
In practice, inequality (a) can be satisfied for an 
25 area that is not close to the edge 94, if there is a 



- 148 - 

S04P0831 

variation in data due to, for example, a fine pattern of a 
subject. To avoid the above problem and to make a more 
reliable determination, the calculation circuit 24 
determines whether there is a vertical edge 94 by 
5 determining whether the following inequality including 
variances SGO and SGO 1 as additional terms is satisfied, 
that is, the calculation circuit 24 determines that there is 
a vertical edge when inequality (b) shown below is satisfied. 
EGO + SGO < EGO" < EGO' - SGO' 
10 or 

EGO* + SGO 1 < EGO" < EGO - SGO (b) 
Use of inequality (b) makes it possible to prevent an 
area in which pixel values have variations due to a fine 
pattern or the like and thus in which variances SGO and SGO 1 
15 are large from being incorrectly identified as an area in 
which there is an edge. 

A method of determining whether there is a horizontal 
edge is described below with reference to Fig. 29. In Fig. 
29, similar parts to those in Fig. 28 are denoted by similar 
20 reference numerals, and a duplicated description thereof is 
not given herein. 

To determine whether there is a horizontal edge 104, 
the calculation circuit 24 determines the mean value and the 
variance of pixel values Gobs(k, ig, jg) at positions 
25 denoted by open circles in an upper edge- judgment region 101 
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extending in a horizontal direction and including a region 
A00, a region A10, and a region A20 (hereinafter, the region 
101 will be referred to simply as the upper region 101) , the 
mean value and the variance of pixel values Gobs(k, ig, jg) 
5 at positions denoted by open circles in a central edge- 
judgment region 102 extending in a horizontal direction and 
including a region A01, a region All, and a region A21 
(hereinafter, the region 102 will be referred to simply as 
the central region 102) , and the mean value and the variance 

10 of pixel values Gobs(k, ig, jg) at positions denoted by open 
circles in a lower edge- judgment region 103 extending in a 
horizontal direction and including a region A02, a region 
A12, and a region A22 (hereinafter, the region 103 will be 
referred to simply as the lower region 103) . 

15 That is, the calculation circuit 24 determines the mean 

value EG1 of pixel values Gobs(k, ig, jg) at positions 
denoted by open circles 81-1 to 81-3, 81-10, 82-1, 82-2, and 
82-9 in an upper region 101 and also determines the variance 
SGI of those pixel values Gobs(k, ig, jg) . The calculation 

20 circuit 24 also determines the mean value EG1 ' 1 of pixel 

values Gobs(k, ig, jg) at positions denoted by open circles 
81-4 to 81-6, 82-3 to 82-5, and 83 in a central region 102. 
Furthermore, the calculation circuit 24 determines the mean 
value EG1 1 of pixel values Gobs(k, ig, jg) at positions 

25 denoted by open circles 81-7 to 81-9 and 82-6 to 82-8 in a 
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lower region 103 and also determines the variance SGI' of 
those pixel values Gobs(k, ig, jg) . 

By way of example, if there is a horizontal edge 104 
close to the position of interest denoted by the open circle 
83, the following inequality (c) holds for the mean value 
EG1 of the upper region 101, the mean value EG1 1 ' of the 
central region 102, and the mean value EG1 1 of the lower 
region 103 taking into account variations in data due to a 
fine pattern of a subject, and thus the calculation circuit 
24 determines that there is a horizontal edge when 
inequality (c) is satisfied. 

EG1 + SGI < EG1 " < EG1 ' -SGI 1 

or 

EG1 ' + SGI' < EG1 " < EG1 - SGI (c) 

Use of inequality (c) makes it possible to prevent an 
area in which pixel values have variations due to a fine 
pattern or the like and thus in which variances SGI and SGI 1 
are large from being incorrectly identified as an area in 
which there is an edge. Note that a determination as to 
whether there is a horizontal edge may be made based on 
inequality similar to (a) including only the mean values 
without taking into account variances. 

A method of determining whether there is an edge 
extending from upper left to lower right is described below 
with reference to Fig. 30. In Fig. 30, similar parts to 
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those in Fig. 28 are denoted by similar reference numerals, 
and a duplicated description thereof is not given herein. 

To determine whether there is an edge 114 extending 
from upper left to lower right, the calculation circuit 24 
5 determines the mean value and the variance of pixel values 
Gobs(k, ig, jg) at positions denoted by open circles in a 
lower- left edge- j udgment region 111 including a region A01, 
a region A02, and a region A12 (hereinafter, the region 111 
will be referred to simply as the lower left region 111) , 

10 the mean value and the variance of pixel values Gobs(k, ig, 
jg) at positions denoted by open circles in a central edge- 
judgment region 112 including a region A00, a region All, 
and a region A22 (hereinafter, the region 112 will be 
referred to simply as the central region 112) , and the mean 

15 value and the variance of pixel values Gobs(k, ig, jg) at 
positions denoted by open circles in an upper-left edge- 
judgment region 113 including a region A10, a region A20, 
and a region A21 (hereinafter, the region 113 will be 
referred to simply as the upper left region 113) . 

20 That is, the calculation circuit 24 determines the mean 

value EG2 of pixel values Gobs(k, ig, jg) at positions 
denoted by open circles 81-9 and 82-3 to 82-8 in a lower- 
left region 111 and also determines the variance SG2 of 
those pixel values Gobs(k, ig, jg) . The calculation circuit 

25 24 also determines the mean value EG2 • ' of pixel values 
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Gobs(k, ig, jg) at positions denoted by open circles 81-7, 
81-8, 82-1, 82-2, and 83 in a central region 112. 
Furthermore, the calculation circuit 24 determines the mean 
value EG2 ■ of pixel values Gobs(k, ig, jg) at positions 
5 denoted by open circles 81-1 to 81-6, 81-10, and 82-9 in an 
upper right region 113 and also determines the variance SG2 1 
of those pixel values Gobs(k, ig, jg) . 

By way of example, if there is an edge 114 extending 
from upper left to lower right at a location close to the 
10 position of interest denoted by the open circle 83, the 

following inequality (d) holds for the mean value EG2 of the 
left lower 111, the mean value EG2 ' 1 of the central region 
112, and the mean value EG2 1 of the upper right region 113, 
taking into account variations in data due to a fine pattern 
15 or the like of a subject, and thus the calculation circuit 
24 determines that there is an edge extending from upper 
left to lower right when inequality (d) is satisfied. 

EG2 + SG2 < EG2 ' « < EG2 ■ - SG2 ' 

or 

20 EG2 ' + SG2 ' < EG2 11 < EG2 - SG2 (d) 

Use of inequality (d) makes it possible to prevent an 
area in which pixel values have variations due to a fine 
pattern or the like and thus in which variances SG2 and SG2 ' 
are large from being incorrectly identified as an area in 

25 which there is an edge. Note that a determination as to 
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whether there is an edge extending from upper left to lower 
right may be made based on inequality similar to (a) 
including only the mean values without taking into account 
variances . 

5 A method of determining whether there is an edge 

extending from upper right to lower left is described below 
with reference to Fig. 31. In Fig. 31, similar parts to 
those in Fig. 28 are denoted by similar reference numerals, 
and a duplicated description thereof is not given herein. 

10 To determine whether there is an edge 124 extending 

from upper right to lower left, the calculation circuit 24 
determines the mean value and the variance of pixel values 
Gobs(k, ig, jg) at positions denoted by open circles in an 
upper-left edge- judgment region 121 including a region A00, 

15 a region A01, and a region A10 (hereinafter, the region 121 
will be referred to simply as the upper left region 121) , 
the mean value and the variance of pixel values Gobs(k, ig, 
jg) at positions denoted by open circles in a central edge- 
judgment region 122 including a region A02, a region All, 

20 and a region A20 (hereinafter, the region 122 will be 

referred to simply as the central region 122) , and the mean 
value and the variance of pixel values Gobs(k, ig, jg) at 
positions denoted by open circles in a lower-right edge- 
judgment region 123 including a region A12, a region A21, 

25 and a region A22 (hereinafter, the region 123 will be 
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referred to simply as the lower right region 123) . 

That is, the calculation circuit 24 determines the mean 
value EG3 of pixel values Gobs (k, ig, jg) at positions 
denoted by open circles 81-10, 82-1 to 82-5, and 82-9 in an 
5 upper left 121 and also determines the variance SG3 of those 
pixel values Gobs(k, ig, jg) . The calculation circuit 24 
also determines the mean value EG3 1 1 of pixel values Gobs(k, 
ig, jg) at positions denoted by open circles 81-1 to 81-3, 
82-6, 82-7, and 83 in a central region 122. Furthermore, 
10 the calculation circuit 24 determines the mean value EG3 1 of 
pixel values Gobs(k, ig, jg) at positions denoted by open 
circles 81-4 to 81-9 and 82-8 in a lower right region 123 
and also determines the variance SG3 ■ of those pixel values 
Gobs (k, ig, jg) . 

15 By way of example, if an edge 124 extending from upper 

left to lower right is located close to the position of 
interest denoted by the open circle 83, the following 
inequality (e) holds for the mean value EG3 of the upper 
left 121, the mean value EG3 ' ' of the central region 122, 

20 and the mean value EG3 ' of the lower right 123 taking into 
account variations in data due to a fine pattern of a 
subject, and thus the calculation circuit 24 determines that 
there is an edge extending from upper left to lower right 
when inequality (e) is satisfied. 

2 5 EG3 + SG3 < EG3 " < EG3 ' -SG3 » 
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or 

EG3» + SG3 ' < EG3" < EG3 - SG3 (e) 

Use of inequality (e) makes it possible to prevent an 
area in which pixel values have variations due to a fine 
5 pattern or the like and thus in which variances SGI and SGI ' 
are large from being incorrectly identified as an area in 
which there is an edge. Note that a determination as to 
whether there is an edge extending from upper right to lower 
left may be made based on inequality similar to (a) 
10 including only the mean values without taking into account 
variances . 

As described above with reference to Figs. 28 to 31, 
the calculation circuit 24 determines whether there is an 
edge in each of four directions, the vertical direction, the 

15 horizontal direction, the direction from upper left to lower 
right, and the direction from upper right to lower left, by 
determining whether inequalities described above are 
satisfied. Note that the calculation circuit 24 makes the 
determination as to whether there is an edge not only for 

20 the green color but also red and blue colors. 

There is a possibility that inequalities described 
above are satisfied in for a plurality of directions of the 
above four directions. Because there can exist only one 
true edge, and thus only one most dominant edge is employed 

25 and the other edges are discarded. 
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More specifically, for example, the calculation circuit 
24 calculates the following value in each of the directions 
in which edges are detected 

| (EGm 1 - EGm) / (SGm' + SGm) | 
5 where m = 0 to 3 , and the calculation circuit 24 determines 
m corresponding to the greatest value. If the value is 
greatest when m = 0, a vertical edge is determined to be 
dominant. If m = 1, a horizontal edge is determined to be 
dominant, If m = 2, an edge extending in a direction from 
10 upper left to lower right is determined to be dominant. If 
m = 3, an edge extending in a direction from lower left to 
lower left is determined to be dominant. Note that in the 
above equation, |x| denotes the absolute value of x. 

Depending on the direction of the edge detected at the 
15 position of interest, the calculation circuit 24 performs a 
special process described below on the pixel value Gobs(k, 
ig# jg) observed at the position of interest in the edge 
part . 

That is, in the special process, when pixel values 
20 Gobs(k, ig, jg) are substituted into the spring equation 
(25) associated with green light intensities, the pixel 
value Gobs(k, ig, jg) is changed for the pixel value Gobs(k, 
ig / jg) observed at the position in the edge part. 

The changing of the pixel value Gobs(k, ig, jg) to be 
25 substituted is performed, for example, in accordance to a 
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first or second method described above. 

In the first method, when an edge is detected, for 
example, at a position of interest denoted by an open circle 
83 in Figs. 28 to 31, the pixel value Gobs (k, ig, jg) at 
5 that position is discarded. That is, the pixel value Gobs(k, 
i^/ jg) a t the position at which the edge is detected is not 
incorporated into the spring equation (25) associated with 
the green light intensity. In other words, a spring that 
would pull the light intensity in a wrong direction is 

10 removed from the spring model shown in Fig. 22 (Fig. 23), 
and thus the value of Lg ( I ' , J ! ) can be. determined more 
accurately (and thus a shaper image can be obtained) . 

In the second method, the pixel value Gobs(k, ig, jg) 
at the position of interest, for example, denoted by an open 

15 circle 83 in Figs. 28 to 31 is replaced with a value 

calculated from pixel values Gobs(k, ig, jg) observed in 
nine regions A0 0 to A2 2 included in the edge judgment region 
defined at the position of interest in the edge detection 
process described above, and the resultant replaced value is 

20 substituted into the spring equation (25) associated with 
the green light intensity. 

The second method is described in further detail below. 
Fig. 32 shows a region All including a position of 
interest denoted by an open circle 83 in a state in which a 

25 vertical edge shown in Fig. 28 is detected. In Fig. 32, one 
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axis is defined in a X direction in the reference coordinate 
system, a Y direction in the reference coordinate system is 
defined in a direction perpendicular to the X direction, and 
an axis perpendicular to both the X direction and the Y 
5 direction represents pixel values. 

In the example shown in Fig. 32, the mean values EGO, 
EGO 1 , and EGO ' 1 and the variances SGO and SGO 1 determined 
for the edge judgment region satisfy the condition EGO' + 
SGO' < EGO' 1 < EGO - SGO, and thus there is a vertical edge. 

10 In this case, the true green light intensity Lg(I - 0.5, 

J - 0.5) at a- position (I - 0.5, J - 0.5) denoted by a solid 
circle 85 and the true green light intensity Lg ( I - 0.5, J + 
0.5) at a position (I - 0.5, J + 0.5) denoted by a solid 
circle 87 are greater than the pixel value Gobs(k, ig, jg) 

15 observed at a position (x, y) of interest denoted by the 

open circle 83. The difference between the true green light 
intensity Lg ( I - 0.5, J - 0.5) at the position (I - 0.5, J - 
0.5) denoted by the solid circle 85 and the pixel value 
Gobs(k, ig, jg) at the position (x, y) denoted by the open 

20 circle 83 and the difference between the true green light 

intensity Lg(I - 0.5, J + 0.5) at the position (I - 0.5, J + 
0.5) denoted by the solid circle 87 and the pixel value 
Gobs(k, ig, jg) at the position (x, y) denoted by the open 
circle 83 depend on the difference between the mean values 

25 EGO and EGO * , that is, EGO - EGO'. 
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The true green light intensity Lg(I + 0.5, J - 0.5) at 
a position (I + 0.5, J - 0.5) denoted by a solid circle 84 
and the true green light intensity Lg ( I + 0.5, J + 0.5) at a 
position (I + 0.5, J + 0.5) denoted by a solid circle 86 are 
5 greater than the pixel value Gobs(k, ig, jg) observed at the 
position (x, y) of interest denoted by the open circle 83. 
The difference between the true green light intensity Lg(I + 
0.5, J - 0.5) at the position (I + 0.5, J - 0.5) denoted by 
the solid circle 84 and the pixel value Gobs (k, ig, jg) at 

10 the position (x, y) denoted by the open circle 83 and the 

difference between the true green light intensity Lg ( I + 0.5, 
J + 0.5) at the position (I + 0.5, J + 0.5) denoted by the 
solid circle 86 and the pixel value Gobs(k, ig, jg) at the 
position (x, y) denoted by the open circle 83 depend on the 

15 difference between the mean values EGO and EGO 1 , that is, 
EGO - EGO 1 . 

In view of the above, the calculation circuit 24 
determines a plane Ql passing through the pixel value Gobs(k, 
ig, jg) at the position (x, y) denoted by the open circle 83 

20 and having a slope p in the X direction, where p = (EGO - 
EGO 1 ) / 2 . The slope p is determined by dividing the 
difference between the mean value EGO of pixel values in the 
left region 91 in Fig. 28 and the mean value EGO* of pixel 
values in the right region 93 by the center- to-center 

25 distance in the X direction between the left region 91 and 
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the right region 92, that is, by 2 . Note that the slope p 
indicates the degree of change in light intensity in a 
vertical edge part in a direction perpendicular to the edge, 
that is, the slope p indicates the slope of the edge. 
5 Furthermore, the calculation circuit 24 calculates the 

value (pixel value) at the position (I - 0.5, J - 0.5) 
denoted by the solid circle 85 in the plane Ql . The 
calculated pixel value is denoted by Gobs0(k, ig, jg) . 
Furthermore, the calculation circuit 24 calculates the value 

10 (pixel value) at the position (I - 0.5, J + 0.5) denoted by 
the solid circle 87 in the plane Ql The calculated pixel 
value is denoted by Gobsl(k, ig, jg) . Similarly, the 
calculation circuit 24 calculates the value (pixel value) at 
the position (I + 0.5, J - 0.5) denoted by the solid circle 

15 84 in the plane Ql and the value (pixel value) at the 

position (I + 0.5, J + 0.5) denoted by the solid circle 86 
in the plane Ql . The resultant calculated pixel values are 
denoted as Gobs2 (k, ig, jg) and Gobs3 (k, ig, jg) , 
re spec t i ve ly . 

20 Because the plane Ql has a slope p only in the X 

direction, the pixel value GobsO (k, ig, jg) and the pixel 
Gobsl(k, ig, jg) are equal to each other, and the pixel 
value Gobs2 (k, ig, jg) and the pixel Gobs3 (k, ig, jg) are 
equal to each other. 

25 In the third embodiment described above, as shown in 
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Fig. 22, equation (25) is written based on the spring model 
representing the equilibrium among the tree green intensity 
Lg(I', J') at the position (I', J 1 ) and pixel values Gobs(k, 
19/ jg) observed in the vicinity of the position (I 1 , J ! ). 
5 Herein, the pixel values Gobs(k, ig, jg) observed in the 
vicinity of the position (I 1 , J') refer to those pixel 
values Gobs(k, ig, jg) whose (k, ig, jg) satisfies I 1 - 1 < 
x < I 1 +1 and J 1 - 1 < y < J 1 +1 with respect to the 
converted position (x, y) in the reference coordinate system 

10 obtained by converting, based on conversion parameters (0k, 
Tkx, Tky, Sk) , the position (ig - 0.5, jg - 0.5) in the 
coordinate system of the k-th image. 

In Fig. 32, for example, when the position (I + 0.5, J 
- 0.5) denoted by the solid circle 84 is selected as the 

15 potion (I', J') of interest, the pixel value Gobs(k, ig, jg) 
at the position (x, y) denoted by the open circle 83 is 
included in a group of pixel values Gobs(k, ig, jg) observed 
in the vicinity of the position denoted by the solid circle 
84. Therefore, in the third embodiment described above, the 

20 pixel value Gobs(k, ig, jg) at the position (x, y) denoted 
by the open circle 83 is substituted into equation (25) . 

In contrast, in the sixth embodiment, instead of the 
pixel value Gobs(k, ig, jg) at the position (x, y) denoted 
by the open circle 83, the calculated value, that is, the 

25 pixel value Gobs2 (k, ig, jg) at the position (1+0.5, J - 
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0.5) denoted by the solid circle 84 in the plane Ql is 
substituted into equation (25) . 

When the position (I - 0.5, J - 0.5) denoted by the 
solid circle 85 is selected as the potion (I 1 , J') of 
5 interest, instead of the pixel value Gobs(k, ig, jg) at the 
position (x, y) denoted by the open circle 83, the 
calculated pixel value GobsO (k, ig, jg) at the position (I - 
0.5, J - 0.5) denoted by the solid circle 85 in the plane Ql 
is substituted into equation (25) . 

10 In a case in which the position (1+0.5, J + 0.5) 

denoted by the solid circle 86 is selected as the potion (I', 
J') of interest, instead of the pixel value Gobs(k, ig, jg) 
at the position (x, y) denoted by the open circle 83, the 
calculated pixel value Gobs3 (k, ig, jg) at the position (I + 

15 0.5, J + 0.5) denoted by the solid circle 86 in the plane Ql 
is substituted into equation (25) . 

When the position (I - 0.5, J + 0.5) denoted by the 
solid circle 87 is selected as the potion (I', J') of 
interest, instead of the pixel value Gobs(k, ig, jg) at the 

20 position (x, y) denoted by the open circle 83, the 

calculated pixel value Gobsl(k, ig, jg) at the position (I - 
0.5, J + 0.5) denoted by the solid circle 87 in the plane Ql 
is substituted into equation (25) . 

As described above, when a pixel value Gobs(k, ig, jg) 

25 (such as that at the position denoted by the open circle 83 
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in Fig. 32) is observed in a vertical edge part (where there 
is an abrupt change in green light intensity) , the pixel 
value is changed (corrected) to a pixel value (GobsO (k, ig, 
jg) , GobsKk, ig, jg) , Gobs2 (k, ig, jg) , or Gobs3 (k, ig, 
5 jg) ) depending on the slope p of the edge (the degree of 
change in pixel value) , the changed pixel value is 
substituted into equation (25). As a result, in the spring 
model, the object V indicating Lg ( I ' , J') shown in Fig. 23 
is pulled to a correct position, and thus the obtained value 

10 Lg(I', J') is more likely to be the true green light 
intensity (and thus a shaper image can be obtained) . 

Fig. 33 shows the region All including the position of 
interest denoted by then open circle 83 in a state in which 
a horizontal edge shown in Fig. 29 is detected. In Fig. 33, 

15 one axis is defined in a X direction in the reference 

coordinate system, a Y direction in the reference coordinate 
system is defined in a direction perpendicular to the X 
direction, and an axis perpendicular to both the X direction 
and the Y direction represents pixel values. 

2 0 In the example shown in Fig. 33, the mean values EG1, 

EG1 ' , and EG1 ■ 1 and the variances SGI and SGI ■ determined 
for the edge judgment region satisfy the condition EG1 ' + 
SGI* < EG1 ' 1 < EG1 - SGI, and thus there is a horizontal 
edge . 

25 In this case, the true green light intensity Lg(I + 0.5, 
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J - 0.5) at a position (I + 0.5, J - 0.5) denoted by the 
solid circle 84 and the true green light intensity Lg(I - 
0.5, J - 0.5) at the position (I - 0.5, J - 0.5) denoted by 
the solid circle 85 are greater than the pixel value Gobs(k, 
5 ig/ jg) observed at a position (x, y) of interest denoted by 
the open circle 83 . The difference between the true green 
light intensity Lg(I + 0.5, J - 0.5) at the position (I + 
0.5, J - 0.5) denoted by the solid circle 84 and the pixel 
value Gobs(k, ig, jg) at the position (x, y) denoted by the 

10 open circle 83 and the difference between the true green 
light intensity Lg(I - 0.5, J - 0.5) at the position (I - 
0.5, J - 0.5) denoted by the solid circle 85 and the pixel 
value Gobs(k, ig, jg) at the position (x, y) denoted by the 
open circle 83 depend on the difference between the mean 

15 values EG1 and EG1 1 , that is, EG1 - EG1 ' . 

The true green light intensity Lg(I + 0.5, J + 0.5) at 
a position (1+0.5, J+0.5) denoted by a solid circle 86 
and the true green light intensity Lg ( I - 0.5, J + 0.5) at a 
position (1-0.5, J+0.5) denoted by a solid circle 87 are 

20 greater than the pixel value Gobs(k, ig, jg) observed at the 
position (x, y) of interest denoted by the open circle 83. 
The difference between the true green light intensity Lg ( I + 
0.5, J + 0.5) at the position (I + 0.5, J + 0.5) denoted by 
the solid circle 86 and the pixel value Gobs(k, ig, jg) at 

25 the position (x, y) denoted by the open circle 83 and the 
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difference between the true green light intensity Lg(I - 0.5, 
J + 0.5) at the position (I - 0.5, J + 0.5) denoted by the 
solid circle 87 and the pixel value Gobs(k, ig, jg) at the 
position (x, y) denoted by the open circle 83 depend on the 
5 difference between the mean values EG1 and EG1 1 , that is, 
EG1 - EG1 ' . 

In view of the above, the calculation circuit 24 
determines a plane Q2 passing through the pixel value Gbbs(k, 
i^/ jg) at the position (x, y) denoted by the open circle 83 

10 and having a slope p in the Y direction, where p = (EG1 - 
EG01) / 2. The slope p is determined by dividing the 
difference between the mean value EG1 of pixel values in the 
upper region 101 in Fig. 29 and the mean value EG1 ' of pixel 
values in the lower region 103 by the center- to-center 

15 distance in the Y direction between the upper region 101 and 
the lower region 103, that is, by 2. Note that the slope p 
indicates the degree of change in light intensity in a 
horizontal edge part in a direction perpendicular to the 
edge , that is , the slope p indicates the slope of the edge . 

20 Furthermore, the calculation circuit 24 calculates the 

value (pixel value) at the position (I - 0.5, J - 0.5) 
denoted by the solid circle 85 in the plane Q2 . The 
calculated pixel value is denoted by GobsO (k, ig, jg) . 
Furthermore, the calculation circuit 24 calculates the value 

25 (pixel value) at the position (I - 0.5, J + 0.5) denoted by 
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the solid circle 87 in the plane Q2 . The calculated pixel 
value is denoted by Gobsl(k, ig, jg) . Similarly, the 
calculation circuit 24 calculates the value (pixel value) at 
the position (I + 0.5, J - 0.5) denoted by the solid circle 
5 84 in the plane Q2 and the value (pixel value) at the 

position (1+0.5, J + 0.5) denoted by the solid circle 86 
in the plane Q2 . The resultant calculated pixel values are 
denoted as Gobs2 (k, ig, jg) and Gobs3 (k, ig, jg) , 
respectively . 

10 Because the plane Q2 has a slope p only in the Y 

direction, the pixel value Gobs0(k, ig, jg) and the pixel 
Gobs2 (k, ig, jg) are equal to each other, and the pixel 
value Gobsl(k, ig, jg) and the pixel Gobs3 (k, ig, jg) are 
equal to each other. 

15 For the horizontal edge, substitution is performed in a 

similar manner to the vertical edge shown in Fig. 32. That 
is, when the position (I + 0.5, J - 0.5) denoted by the 
solid circle 84 is selected as the potion (I 1 , J') of 
interest, instead of the pixel value Gobs(k, ig, jg) at the 

20 position (x, y) denoted by the open circle 83, the 

calculated pixel value Gobs2 (k, ig, jg) at the position (I + 
0.5, J - 0.5) denoted by the solid circle 84 in the plane Q2 
is substituted into equation (25) . 

When the position (I - 0.5, J - 0.5) denoted by the 

25 solid circle 85 is selected as the potion (I 1 , J') of 
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interest, instead of the pixel value Gobs(k, ig, jg) at the 
position (x, y) denoted by the open circle 83, the 
calculated pixel value GobsO (k, ig, jg) at the position (I - 
0.5, J - 0.5) denoted by the solid circle 85 in the plane Q2 
5 is substituted into equation (25) . 

When the position (I + 0.5, J + 0.5) denoted by the 
solid circle 86 is selected as the potion (I', J 1 ) of 
interest, instead of the pixel value Gobs(k, ig, jg) at the 
position (x, y) denoted by the open circle 83, the 

10 calculated pixel value Gobs3 (k, ig, jg) at the position (I + 
0.5, J + 0.5) denoted by the solid circle 86 in the plane Q2 
is substituted into equation (25) . 

When the position (I - 0.5, J + 0.5) denoted by the 
solid circle 87 is selected as the potion (I', J') of 

15 interest, instead of the pixel value Gobs (k, ig, jg) at the 
position (x, y) denoted by the open circle 83, the 
calculated pixel value Gobsl(k, ig, jg) at the position (x, 
y) denoted by the solid circle 87 in the plane Q2 is 
substituted into equation (25) . 

20 As described above, when a pixel value Gobs(k, ig, jg) 

(such as that at the position denoted by the open circle 83 
in Fig. 33) is observed in a horizontal edge part (where 
there is an abrupt change in green light intensity) , the 
pixel value is changed (corrected) to a pixel value (GobsO (k, 

25 ig, jg) , Gobsl(k, ig, jg) , Gobs2(k, ig, jg), or Gobs3 (k, ig, 
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jg) ) depending on the slope p of the edge (the degree of 
change in pixel value) , the changed pixel value is 
substituted into equation (25) . As a result, in the spring 
model, the object V indicating Lg(I', J 1 ) shown in Fig. 23 
5 is pulled to a correct position, and thus the obtained value 
Lg(I', J') is more likely to be the true green light 
intensity (and thus a shaper image can be obtained) . 

Fig. 34 shows the region All including the position of 
interest denoted by then open circle 83 in a state in which 

10 an edge extending in a direction from upper left to lower 

right shown in Fig. 30 is detected. In Fig. 34, one axis is 
defined in a X direction in the reference coordinate system, 
a Y direction in the reference coordinate system is defined 
in a direction perpendicular to the X direction, and an axis 

15 perpendicular to both the X direction and the Y direction 
represents pixel values . 

In the example shown in Fig. 34, the mean values EG2 , 
EG2 1 , and EG2 1 ' and the variances SG2 and SG2 1 determined 
for the edge judgment region satisfy the condition EG2 ' + 

2 0 SG2 ■ < EG2 ' ■ < EG2 ' - SG2 ' , and thus there is an edge 

extending in the direction from upper left to lower right. 

In this case, the true green light intensity Lg ( I + 0.5, 
J - 0.5) at a position (I + 0.5, J - 0.5) denoted by a solid 
circle 84 is greater than the pixel value Gobs(k, ig, jg) 

25 observed at the position (x, y) of interest denoted by the 
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open circle 83. The difference between the true green light 
intensity Lg(I + 0.5, J - 0.5) at the position (1+0.5, J - 
0.5) denoted by the solid circle 84 and the pixel value 
Gobs(k, ig, jg) at the position (x, y) denoted by the open 
5 circle 83 depend on the difference between the mean values 
EG2 and EG2 1 , that is, EG2 - EG2 ' . 

The true green light intensity Lg(I - 0.5, J + 0.5) at 
a position (I - 0.5, J + 0.5) denoted by a solid circle 87 
is greater than the pixel value Gobs(k, ig, jg) observed at 

10 the position (x, y) of interest denoted by the open circle 
83. The difference between the true green light intensity 
Lg(I - 0.5, J+0.5) at the position (I - 0.5, J + 0.5) 
denoted by the solid circle 87 and the pixel value Gobs(k, 
ig, jg) at the position (x, y) denoted by the open circle 83 

15 depend on the difference between the mean values EG2 1 and 
EG2, that is, EG2 1 - EG2 . 

In view of the above, the calculation circuit 24 
determines a plane Q3 passing through the pixel value Gobs(k, 
jg) at the position (x, y) of interest denoted by the 

2 0 open circle 83 and having a slope p in a diagonal direction 
of the region All, that is, in a direction from the position 
(I + 0.5, J - 0.5) denoted by the solid circle 84 to the 
position (I - 0.5, J + 0.5) denoted by the solid circle 87, 
where p = (EG2 ■ - EG2) / 2^2. The slope p is determined by 

25 dividing the difference between the mean value EG2 1 of pixel 
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values in the upper right region 113 in Fig. 3 0 and the mean 
value EG2 of pixel values in the lower left region 111 by 
2V2, that is, the center- to-center distance in a diagonal 
direction from the position (I + 0.5, J - 0.5) denoted by 
5 the solid circle 84 to the position (I - 0.5, J+0.5) 

denoted by the solid circle 87 between the region A20 in the 
upper right region 113 and the region A02 in the lower left 
region 111. Note that the slope p of an edge part extending 
in the direction from upper left to lower right indicates 

10 the degree of change in light intensity in a direction 

perpendicular to the edge, that is, the slope p indicates 
the slope of the edge. 

Furthermore, the calculation circuit 24 calculates the 
value (pixel value) at the position (1-0.5, J - 0.5) 

15 denoted by the solid circle 85 in the plane Q3 . The 

calculated pixel value is denoted by GobsO (k, ig, jg) . The 
calculation circuit 24 also calculates the value (pixel 
value) at the position (I - 0.5, J + 0.5) denoted by the 
solid circle 87 in the plane Q3 . The calculated pixel value 

20 is denoted by Gobsl (k, ig, jg) . Similarly, the calculation 
circuit 24 calculates the value (pixel value) at the 
position (I + 0.5, J - 0.5) denoted by the solid circle 84 
in the plane Q3 and the value (pixel value) at the position 
(1+0.5, J+0.5) denoted by the solid circle 86 in the 

25 plane Q3 . The resultant calculated pixel values are denoted 
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as Gobs2(k, ig, jg) and Gobs3 (k, ig, jg) , respectively. 

Because the plane Q3 has a slope p only in the diagonal 
direction of the region All, that is, in the direction from 
the position (I + 0.5, J - 0.5) denoted by the solid circle 
5 84 to the position (I - 0.5, J+0.5) denoted by the solid 
circle 87, the pixel value GobsO (k, ig, jg) and the pixel 
Gobs3 (k, ig, jg) are equal to each other. 

For the edge extending in the direction from upper left 
to lower right, substitution is performed in a similar 

10 manner to the vertical edge shown in Fig. 32. That is, when 
the position (I + 0.5, J - 0.5) denoted by the solid circle 
84 is selected as the potion (I', J') of interest, instead 
of the pixel value Gobs(k, ig, jg) at the position (x, y) 
denoted by the open circle 83, the calculated pixel value 

15 Gobs2(k, ig, jg) at the position (I + 0.5, J - 0.5) denoted 
by the solid circle 84 in the plane Q3 is substituted into 
equation (25) . 

When the position (I - 0.5, J - 0.5) denoted by the 
solid circle 85 is selected as the potion (I', J') of 

20 interest, instead of the pixel value Gobs(k, ig, jg) at the 
position (x, y) denoted by the open circle 83, the 
calculated pixel value GobsO (k, ig, jg) at the position (I - 
0.5, J - 0.5) denoted by the solid circle 85 in the plane Q3 
is substituted into equation (25) . 

25 When the position (I + 0.5, J + 0.5) denoted by the 
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solid circle 86 is selected as the potion (I', J 1 ) of 
interest, instead of the pixel value Gobs(k, ig, jg) at the 
position (x, y) denoted by the open circle 83, the 
calculated pixel value Gobs3 (k, ig, jg) at the position (I + 
5 0.5, J + 0.5) denoted by the solid circle 86 in the plane Q3 
is substituted into equation (25) . 

When the position (I - 0.5, J + 0.5) denoted by the 
solid circle 87 is selected as the potion J 1 ) of 

interest, instead of the pixel value Gobs (k, ig, jg) at the 

10 position (x, y) denoted by the open circle 83, the 

calculated pixel value Gobsl (k, ig, jg) at the position (I - 
0.5, J + 0.5) denoted by the solid circle 87 in the plane Q3 
is substituted into equation (25) . 

As described above, when a pixel value Gobs(k, ig, jg) 

15 (such as that at the position denoted by the open circle 83 
in Fig. 34) is observed in an edge part extending in the 
direction from upper left to lower right (where there is an 
abrupt change in green light intensity) , the pixel value is 
changed (corrected) to a pixel value (Gobs0(k, ig, jg) , 

20 Gobsl (k, ig, jg) , Gobs2 (k, ig, jg) , or Gobs3 (k, ig, jg) ) 

depending on the slope p of the edge (the degree of change 
in pixel value) , the changed pixel value is substituted into 
equation (25). As a result, in the spring model, the object 
V indicating Lg(I', J f ) shown in Fig. 23 is pulled to a 

25 correct position, and thus the obtained value Lg ( I ' , J') is 
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more likely to be the true green light intensity (and thus a 
shaper image can be obtained) . 

Fig. 35 shows the region All including the position of 
interest denoted by then open circle 83 in a state in which 
5 an edge extending in a direction from upper right to lower 
left shown in Fig. 31 is detected. In Fig. 34, one axis is 
defined in a X direction in the reference coordinate system, 
a Y direction in the reference coordinate system is defined 
in a direction perpendicular to the X direction, and an axis 

10 perpendicular to both the X direction and the Y direction 
represents pixel values. 

In the example shown in Fig. 35, the mean values EG3 , 
EG3 ' , and EG3 * * and the variances SG3 and SG3 1 determined 
for the edge judgment region satisfy the condition EG3 ' + 

15 SG3 1 < EG3 ' 1 < EG3 - SG3 , and thus there is an edge 

extending in the direction from upper right to lower left. 

In this case, the true green light intensity Lg(I - 0.5, 
J - 0.5) at the position (I - 0.5, J - 0.5) denoted by the 
solid circle 85 is greater than the pixel value Gobs(k, ig, 

20 jg) observed at the position (x, y) of interest denoted by 
the open circle 83. The difference between the true green 
light intensity Lg(I - 0.5, J - 0.5) at the position (I - 
0.5, J - 0.5) denoted by the solid circle 85 and the pixel 
value Gobs(k, ig, jg) at the position (x, y) denoted by the 

25 open circle 83 depend on the difference between the mean 



r 
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values EG3 and EG3 ' , that is, EG3 - EG3 ' . 

The true green light intensity Lg(I + 0.5, J + 0.5) at 
the position (1+0.5, J+0.5) denoted by the solid circle 
86 is greater than the pixel value Gobs(k, ig, jg) observed 
5 at the position (x, y) of interest denoted by the open 
circle 83 . The difference between the true green light 
intensity Lg ( I + 0.5, J + 0.5) at the position (1+0.5, J + 
0.5) denoted by the solid circle 86 and the pixel value 
Gobs (k, ig, jg) at the position (x, y) denoted by the open 

10 circle 83 depend on the difference between the mean values 
EG3 and EG3 ' , that is, EG3 - EG3 1 . 

In view of the above, the calculation circuit 24 
determines a plane Q4 passing through the pixel value Gobs(k, 
ig, jg) at the position (x, y) of interest denoted by the 

15 open circle 83 and having a slope p in a diagonal direction 
of the region All, that is, in a direction from the position 
(I - 0.5, J - 0.5) denoted by the solid circle 85 to the 
position (I + 0.5, J + 0.5) denoted by the solid circle 86, 
where p = (EG3 - EG3 ' ) / 2^2. The slope p is determined by 

20 dividing the difference between the mean value EG3 of pixel 
values in the upper left region 121 in Fig. 31 and the mean 
value EG3 ■ of pixel values in the lower right region 123 by 
2^2, that is, the center- to-center distance in a diagonal 
direction from the position (I - 0.5, J - 0.5) denoted by 

25 the solid circle 85 to the position (1+0.5, J+0.5) 
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denoted by the solid circle 86 between the region A00 in the 
upper left region 121 and the region A22 in the lower right 
region 123. Note that the slope p of an edge part extending 
in the direction from upper right to lower left indicates 
5 the degree of change in light intensity in a direction 

perpendicular to the edge, that is, the slope p indicates 
the slope of the edge. 

Furthermore, the calculation circuit 24 calculates the 
value (pixel value) at the position (1-0.5, J - 0.5) 

10 denoted by the solid circle 85 in the plane Q4 . The 

calculated pixel value is denoted by Gobs0(k, ig, jg) . The 
calculation circuit 24 also calculates the value (pixel 
value) at the position (I - 0.5, J + 0.5) denoted by the 
solid circle 87 in the plane Q4 . The calculated pixel value 

15 is denoted by Gobsl(k, ig, jg) . Similarly, the calculation 
circuit 24 calculates the value (pixel value) at the 
position (I + 0.5, J - 0.5) denoted by the solid circle 84 
in the plane Q4 and the value (pixel value) at the position 
(I + 0.5, J + 0.5) denoted by the solid circle 86 in the 

20 plane Q4 . The resultant calculated pixel values are denoted 
as Gobs2 (k, ig, jg) and Gobs3 (k, ig, jg) , respectively. 

Because the plane Q4 has a slope p only in the diagonal 
direction of the region All, that is, in the direction from 
the position (I - 0.5, J - 0.5) denoted by the solid circle 

25 85 to the position (I + 0.5, J + 0.5) denoted by the solid 
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circle 86, the pixel value Gobsl (k, ig, jg) and the pixel 
Gobs2(k, ig, jg) are equal to each other. 

For the edge extending in the direction from upper 
right to lower left, substitution is performed in a similar 
5 manner to the vertical edge shown in Fig . 32 . That is , when 
the position (I + 0.5, J - 0.5) denoted by the solid circle 
84 is selected as the potion (I 1 , J 1 ) of interest, instead 
of the pixel value Gobs(k, ig, jg) at the position (x, y) 
denoted by the open circle 83, the calculated pixel value 

10 Gobs2 (k, ig, jg) at the position (I + 0.5, J - 0.5) denoted 
by the solid circle 84 in the plane Q4 is substituted into 
equation (25) . 

When the position (I - 0.5, J - 0.5) denoted by the 
solid circle 85 is selected as the potion (I 1 , J') of 

15 interest, instead of the pixel value Gobs (k, ig, jg) at the 
position (x, y) denoted by the open circle 83, the 
calculated pixel value GobsO (k, ig, jg) at the position (I - 
0.5, J - 0.5) denoted by the solid circle 85 in the plane Q4 
is substituted into equation (25) . 

20 When the position (I + 0.5, J + 0.5) denoted by the 

solid circle 86 is selected as the potion (I 1 , J') of 
interest, instead of the pixel value Gobs (k, ig, jg) at the 
position (x, y) denoted by the open circle 83, the 
calculated pixel value Gobs3 (k, ig, jg) at the position (I + 

25 0.5, J + 0.5) denoted by the solid circle 86 in the plane Q4 
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is substituted into equation (25) . 

When the position (I - 0.5, J + 0.5) denoted by the 
solid circle 87 is selected as the potion (I 1 , J') of 
interest, instead of the pixel value Gobs(k, ig, jg) at the 
5 position (x, y) denoted by the open circle 83, the 

calculated pixel value Gobsl (k, ig, jg) at the position (I - 
0.5, J + 0.5) denoted by the solid circle 87 in the plane Q4 
is substituted into equation (25) . 

As described above, when a pixel value Gobs(k, ig, jg) 

10 (such as that at the position denoted by the open circle 83 
in Fig. 35) is observed in an edge part extending in the 
direction from upper right to lower left (where there is an 
abrupt change in green light intensity) , the pixel value is 
changed (corrected) to a pixel value (GobsO (k, ig, jg) , 

15 Gobsl (k, ig, jg) , Gobs2 (k, ig, jg) , or Gobs3 (k, ig, jg) ) 

depending on the slope p of the edge (the degree of change 
in pixel value) , the changed pixel value is substituted into 
equation (25) . As a result, in the spring model, the object 
V indicating Lg(I', J') shown in Fig. 23 is pulled to a 

20 correct position, and thus the obtained value Lg(I ! , J') is 
more likely to be the true green light intensity (and thus a 
shaper image can be obtained) . 

From the above discussion, in this sixth embodiment, a 
spring equation associated with the green light intensity 

25 corresponding to equation (25) according to the third 
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embodiment is given by equation (34) shown below. Note that 
the true red light intensity Lr(I' # J') and the true blue 
light intensity Lb ( I 1 , J') can be determined in a similar 
manner to the true green light intensity Lg(r , J ! ). 

V2-F((x.y). d'.J')) 



x {Gobs' (k, ig, jg)-Lg(I'.J')} 



= 0 



(Mk x E) 

• • • (34) 

For the given position (I 1 , J 1 ), the sum £ in equation 
(34) is taken for all combinations (k, ig, jg) that satisfy 
I' - 1 < x < I 1 +1 and J' - 1 < y < J' +1 with respect to 
10 the position (x, y) in the reference coordinate system 

obtained by converting, based on conversion parameters (0k, 
Tkx, Tky, Sk) , the position (ig - 0.5, jg - 0.5) on the k-th 
image . 

Note that when one of a vertical edge, a horizontal 
15 edge, an edge extending from upper left to lower right, and 
an edge extending from upper right to lower left is detected 
at a position of a pixel value Gobs(k, ig, jg) of one of 
combinations of integers (k, ig, jg) included in equation 
(34), a calculated pixel value at a position (I', J') in a 
20 plane passing through that pixel value Gobs(k, ig, jg) of (k, 
i?/ jg) and having the slope p equal to the slope of the 
edge is substituted into Gobs' (k, ig, jg) of equation (34) 
instead of the pixel value Gobs(k, ig, jg) of (k, ig, jg) . 
When none of a vertical edge, a horizontal edge, an edge 



- 179 - 

S04P0831 

extending from upper left to lower right, and an edge 
extending from upper right to lower left is detected at a 
position of a pixel value Gobs(k, ig, jg) of any of 
combinations of integers (k, ig, jg) included in equation 
(34), the pixel value Gobs(k, ig, jg) of (k, ig, jg) is 
directly substituted into Gobs' (k, ig, jg) in equation (34). 

Furthermore, in this sixth embodiment, a spring 
equation associated with the green light intensity and a 
spring equation associated with the blue light intensity 
corresponding to equations (26) and (27) according to the 
third embodiment are is given by equations (35) and (36) 
shown below. 

V2-F((x.y), (I', J')) 



x {Robs'(k, ir, jr)-Lr(I'.J')} 



= 0 



(Mk x E) 

s 

• • • (35) 

For the given position (I', J'), the sum S in equation 
(35) is taken for all combinations (k, ir, jr) that satisfy 
I' - 1 < x < I' +1 and J ' -l<y<J'+l with respect to 
the position (x, y) in the reference coordinate system 
obtained by converting, based on conversion parameters (0k, 
Tkx, Tky, Sk) , the position (ir - 0.5, jr - 0.5) on the k-th 
image . 

Note that when one of a vertical edge, a horizontal 
edge, an edge extending from upper left to lower right, and 
an edge extending from upper right to lower left is detected 
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at a position of a pixel value Robs(k, ir, jr) of one of 
combinations of integers (k, ir, jr) included in equation 

(35), a calculated pixel value at a position (I', J') in a 
plane passing through that pixel value Robs(k, ir, jr) of (k, 
5 ir, jr) and having the slope p equal to the slope of the 
edge is substituted into Robs' (k, ir, jr) in equation (35) 
instead of the pixel value Robs(k, ir, jr) of (k, ir, jr) . 
When none of a vertical edge, a horizontal edge, an edge 
extending from upper left to lower right, and an edge 
10 extending from upper right to lower left is detected at a 
position of a pixel value Gobs (k, ig, jg) of any of 
combinations of integers (k, ig, jg) included in equation 

(35), the pixel value Robs(k, ir, jr) of (k, ir, jr) is 
directly substituted into Robs' (k, ir, jr) in equation (35) . 

V2-F((x.y). CI'.J')) 



15 



x {Bobs'(k, ib, jb) -Lb(I\ J')} 



= 0 



(Mk x E) 

• • • (36) 

For the given position (I', J'), the sum Z in equation 
(36) is taken for all combinations (k, ib, jb) that satisfy 
I 1 - 1 < x < I' +1 and J' - 1 < y < J' +1 with respect to 
20 the position (x, y) in the reference coordinate system 

obtained by converting, based on conversion parameters (0k, 
Tkx, Tky, Sk) , the position (ib - 0.5, jb - 0.5) on the k-th 
image . 

Note that when one of a vertical edge, a horizontal 
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edge, an edge extending from upper left to lower right, and 
an edge extending from upper right to lower left is detected 
at a position of a pixel value Bobs(k, ib, jb) of one of 
combinations of integers (k, ib, jb) included in equation 
5 (36), a calculated pixel value at a position (I 1 , J 1 ) in a 

plane passing through that pixel value Bobs(k, ib, jb) of (k, 
ib, jb) and having the slope equal to the slope of the edge 
is substituted into Bobs' (k, ib, jb) of equation (36) 
instead of the pixel value Bobs (k, ib, jb) of (k, ib, jb) . 

10 When none of a vertical edge, a horizontal edge, an edge 
extending from upper left to lower right, and an edge 
extending from upper right to lower left is detected at a 
position of a pixel value Gobs(k, ig, jg) of any of 
combinations of integers (k, ig, jg) included in equation 

15 (36), the pixel value Bobs (k, ib, jb) of (k, ib, jb) is 

directly substituted into Bobs' (k, ib, jb) in equation (36) . 

Now, referring to a flow chart shown in Fig. 36, a 
correct image estimation process in step S4 shown in Fig. 2 
according to a sixth embodiment of the present invention is 

20 described. 

First, in step S141, the calculation circuit 24 selects 
a position (I', J') in the reference coordinate system 
(hereinafter, such a selected position will be referred to a 
position (I 1 , J 1 ) of interest). Note that the position (I', 

25 J') of interest is at the center (i - 0.5, j - 0.5) of a (i- 
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th, j-th) pixel of the first image taken as the reference 
image . 

Thereafter, the process proceeds from step S141 to step 
S142. In step S142, the calculation circuit 24 determines 
5 all combinations of integers (k, ig, jg) that satisfy 1 1 - 1 
< x < I* + 1 and J 1 - 1 < y < J' +1 with respect to the 
position (x, y) in the reference coordinate system obtained 
by converting, based on conversion parameters (9k, Tkx, Tky, 
Sk) , the center position (ig - 0.5, jg - 0.5) of the pixel 

10 that senses a green component of the k-th image, for the 
position (I', J') of interest of each of first to N-th 
images. Thereafter, the process proceeds to step S143. 

In step S143, the calculation circuit 24 determines 
whether there is a vertical edge, a horizontal edge, an edge 

15 extending from upper left to lower right, or an edge 
extending from upper right to lower left, for each of 
combinations of integers (k, ig, jg) determined in step S142. 
If it is determined in step S143 that there is an edge 
extending in one of the four directions, the process 

20 proceeds to step S144 . In step S144, for each combination 
(k, ig, jg) corresponding to the position at which the edge 
was detected, the calculation circuit 24 produces 
(determines) a plane passing through the pixel Gobs(k, ig, 
jg) and having a slope equal to the slope p of the edge. 

25 The calculation circuit 24 then calculates the value (pixel 
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value) at the position (I 1 , J') of interest in the plane. 
Thereafter, the process proceeds from step S144 to step S145. 

In the case in which it is determined in step S143 that 
there is no edge in any of the four directions, the process 
5 proceeds to step S145. Note that the process also proceeds 
to step S145 after completion of step S144 . In step S145, 
the calculation circuit 24 produces a spring equation in the 
form (34) for green light intensity, using all combinations 
of (k, ig, jg) determined in step S142 . Thereafter, the 

10 process proceeds to step S146. For a pixel value Gobs(k, ig, 
jg) at a position corresponding to (k, ig, jg) in the 
reference coordinate system at which an edge was detected in 
step S143, the calculation circuit 24 employs the value 
(pixel value) at the position (I', J 1 ) in the plane 

15 determined in step S144 as the pixel value Gobs (k, ig, jg) 
of (k, ig, jg) at which the edge was detected, and the 
calculation circuit 24 substitutes it into Gobs' (k, ig, jg) 
in equation (34). However, for the pixel value Gobs(k, ig, 
jg) , if it is determined in step S143 that there is no edge 

20 in any of the four directions at the position in the 

reference coordinate system corresponding to (k, ig, jg) , 
the calculation circuit 24 directly substitutes the pixel 
value Gobs(k, ig, jg) of (k, ig, jg) into Gobs' (k, ig, jg) 
in equation (34) . 

25 In step S146, the calculation circuit 24 determines all 
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combinations of integers (k, ir, jr) that satisfy I' - 1 < x 
< I 1 + 1 and J' - 1 < y < J' +1 with respect to the 
position (x, y) in the reference coordinate system obtained 
by converting, based on conversion parameters (0k, Tkx, Tky, 
5 Sk) , the center position (ir - 0.5, jr - 0.5) of the pixel 
that senses a red component of the k-th image, for the 
position ( I 1 , J f ) of interest of each of first to N-th 
images. Thereafter, the process proceeds to step S147. 
In step S147, the calculation circuit 24 determines 

10 whether there is a vertical edge, a horizontal edge, an edge 
extending from upper left to lower right, or an edge 
extending from upper right to lower left, for each of 
combinations of integers (k, ir, jr) determined in step S146. 
If it is determined in step S147 that there is an edge 

15 extending in one of the four directions, the process 

proceeds to step S148. In step S148, for each combination 
(k, ir, jr) corresponding to the position at which the edge 
was detected, the calculation circuit 24 produces 
(determines) a plane passing through the pixel Robs(k, ir, 

20 jr) and having a slope equal to the slope p of the edge. 

The calculation circuit 24 then calculates the value (pixel 
value) at the position (I 1 , J 1 ) of interest in the plane. 
Thereafter, the process proceeds from step S148 to step S149. 
In the case in which it is determined in step S147 that 

25 there is no edge in any of the four directions, the process 
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proceeds to step S14 9. Note that the process also proceeds 
to step S149 after completion of step S148. In step S149, 
the calculation circuit 24 produces a spring equation in the 
form (35) for red light intensity, using all combinations of 
5 (k, ir, jr) determined in step S146. Thereafter, the 

process proceeds to step S150. For a pixel value Robs(k, ir, 
jr) at a position corresponding to (k, ir, jr) in the 
reference coordinate system at which an edge was detected in 
step S147, the calculation circuit 24 employs the value 

10 (pixel value) at the position (I 1 , J') in the plane 

determined in step S148 as the pixel value Robs(k, ir, jr) 
of (k, ir, jr) at which the edge was detected, and the 
calculation circuit 24 substitutes it into Robs 1 (k, ir, jr) 
in equation (35). However, for the pixel value Robs(k, ir, 

15 jr) , if it is determined in step S147 that there is no edge 
in any of the four directions at the position in the 
reference coordinate system corresponding to (k, ir, jr) , 
the calculation circuit 24 directly substitutes the pixel 
value Robs(k, ir, jr) of (k, ir, jr) into Robs' (k, ir, jr) 

20 in equation (35) . 

In step S150, the calculation circuit 24 determines all 
combinations of integers (k, ib, jb) that satisfy I 1 - 1 < x 
< I 1 + 1 and J' - 1 < y < J ! +1 with respect to the 
position (x, y) in the reference coordinate system obtained 

25 by converting, based on conversion parameters (6k, Tkx, Tky, 
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Sk) , the center position (ib - 0.5, jb - 0.5) of the pixel 
that senses a blue component of the k-th image, for the 
position (I 1 , J') of interest of each of first to N-th 
images. Thereafter, the process proceeds to step S151. 
5 In step S151, the calculation circuit 24 determines 

whether there is a vertical edge, a horizontal edge, an edge 
extending from upper left to lower right, or an edge 
extending from upper right to lower left, for each of 
combinations of integers (k, ib, jb) determined in step S152. 

10 If it is determined in step S151 that there is an edge 
extending in one of the four directions, the process 
proceeds to step S152 . In step S152, for each combination 
(k, ib, jb) corresponding to the position at which the edge 
was detected, the calculation circuit 24 produces 

15 (determines) a plane passing through the pixel Bobs(k, ib, 
jb) and having a slope equal to the slope p of the edge. 
The calculation circuit 24 then calculates the value (pixel 
value) at the position (I', J') of interest in the plane. 
Thereafter, the process proceeds from step S152 to step S153 . 

20 In the case in which it is determined in step S151 that 

there is no edge in any of the four directions, the process 
proceeds to step S153 . Note that the process also proceeds 
to step S153 after completion of step S152 . In step S153, 
the calculation circuit 24 produces a spring equation in the 

25 form (36) for blue light intensity, using all combinations 
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of (k, ib, jb) determined in step S150 . Thereafter, the 
process proceeds to step S154 . For a pixel value Bobs(k, ib, 
jb) at a position corresponding to (k, ib, jb) in the 
reference coordinate system at which an edge was detected in 
5 step S151, employs the value (pixel value) at the position 
(I*, J') in the plane determined in step S152 as the pixel 
value Bobs(k, ib, jb) of (k, ib, jb) at which the edge was 
detected, and the calculation circuit 24 substitutes it into 
Gobs ! (k, ib, jb) in equation (36). However, for the pixel 

10 value Bobs(k, ib, jb) , if it is determined in step S151 that 
there is no edge in any of the four directions at the 
position in the reference coordinate system corresponding to 
(k, ib, jb) , the calculation circuit 24 directly substitutes 
the pixel value Bobs(k, ib, jb) of (k, ib, jb) into Bobs 1 (k, 

15 ib, jb) in equation (36) . 

In step S154, the calculation circuit 24 solves the 
spring equation (34) associated with the red light intensity 
determined in step S145, the spring equation (35) associated 
with the red light intensity determined in step S149, and 

20 the spring equation (36) associated with the blue light 

intensity determined in step S153, which are all in the form 
of a linear equation, thereby determining the true green 
light intensity Lg ( I ' , J'), the true red light intensity 
Lr(r , J 1 ), and the true blue light intensity Lb ( I ' , J 1 ) at 

25 the position (I 1 , J') of interest. Thereafter, the process 



- 188 - 

S04P0831 

proceeds to step S155. 

In step S155, the calculation circuit 24 determines 
whether the above-described process is completed for all 
possible positions (I 1 , J'), that is, whether the true green 
5 light intensity Lg(I', J'), the true red light intensity 
Lr(r, J»), and the true blue light intensity Lb (I', J') 
have been determined by taking the center position of each 
of all pixels of the first image as the position (I 1 , J') of 
interest . 

10 If it is determined in step S155 that the process is 

not completed for all possible positions (I 1 , J') of 
interest, the process returns to step S141 to repeat steps 
S141 to S155. That is, the calculation circuit 24 selects, 
as a new position of interest, a position (I', J 1 ) that has 

15 not yet been selected as the position of interest, and 

determines the true green light intensity Lg ( I ' , J'), the 
true red light intensity Lr ( I ' , J 1 ), and the true blue light 
intensity Lb(I', J 1 ) at the position (I 1 , J') of interest. 

On the other hand in the case in which it is determined 

20 in step S155 that the process is completed by taking all 

possible position (I 1 , J') as the position of interest, the 
process proceeds to step S156. In step S156, the 
calculation circuit 24 estimates a correct image (signal) 
from the true green light intensity Lg ( I ' , J'), the true red 

25 light intensity Lr ( I 1 , J'), and the true blue light 
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intensity Lb(I', J') determined in step S154 . The resultant 
image is supplied as an output image to the D/A converter 9 
or the codec 12, and the current process is exited. For 
example, for the (i-th, j-th) pixel, the calculation circuit 
5 24 estimates a correct image signal thereof from the true 

green light intensity L»g(i - 0.5, j - 0.5) determined as the 
green value (G signal) in step S154, the true red light 
intensity Lr(i - 0.5, j - 0.5) determined as the red value 
(R signal) in step S154, and the true blue light intensity 

10 Lb(i - 0.5, j - 0.5) determined as the blue value (B signal) 
in step S154 . The calculation circuit 24 estimates the 
output image by estimating the image signal for all pixels 
whose center is located at the selected position (I*, J') . 
In the sixth embodiment, as described above, for a 

15 pixel value observed in an edge part, the pixel value is 
changed depending on the slope of the edge (the degree of 
change in pixel value) , and the resultant changed pixel 
value is substituted into a corresponding spring equation. 
This makes it possible to obtain an image more likely to be 

2 0 a correct image. That is, a sharper image can be obtained. 

In the example described above, a pixel value located 
in an edge part is replaced with a pixel value at the 
position (I 1 , J' ) of interest depending on the slope 
(steepness) of the edge. Alternatively, for example, the 

25 pixel value in the edge part may be directly used, and the 
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weight is modified depending on the distance to the position 
(I' # J') of interest so that the pixel value located in the 
edge part does not have a significant effect on the spring 
equation . 

5 In the example described above, the slopes of planes Ql 

to Q4 are determined using pixel values observed within nine 
regions (that is, a region with a size of 3 x 3 pixels) 
centered at the position (x, y) in the reference coordinate 
system. Alternatively, the slopes of planes Ql to Q4 may be 

10 determined using pixel values observed in regions selected 

differently. For example, sixteen regions (tat is, a region 
with a size of 4 x 4 pixels) may be employed. 

Now, a correct image estimation process performed by 
the signal processor 7 according to a seventh embodiment of 

15 the present invention is described. The seventh embodiment 
is a modification to the third embodiment described earlier 
with reference to Fig. 24. 

That is, in the seventh embodiment, in addition to the 
spring equations (25) , (26) , and (27) , the constraint 

20 associated with the color correlation among R, G, and B 

signals is imposed on the process of determining the green 
light intensity Lg(x, y) , the true red light intensity Lr(x, 
y) , and the true blue light intensity Lb(x, y) . 

In a local area of an image, there is a color 

25 correlation among colors of the true green light intensity 
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Lg(x, y) , the true red light intensity Lr(x, y) , and the 
true blue light intensity Lb(x, y) corresponding to light 
originating from a subject and being incident on the imaging 
device 4. Therefore, by imposing the color correlation 
5 constraint on the estimation of a true image, it becomes 

possible to produce a sharper low-noise image more likely to 
be the correct image . 

Referring to Figs. 37 and 38, a specific method of 
determining a color correlation constraint is described 

10 below. In the following description referring to Figs. 37 

and 38, by way of example, the correlation between green and 
red colors will be discussed. 

By way of example, the discussion will be made for a 
green pixel G(jg - 1) (ig - 1) at a (ig-th, jg-th) position 

15 of a k'-th image shown in a lower left part of Fig. 37 a red 
pixel G(jg - 1) (ig - 1) at a (ir-th, jr-th) position of a 
k' ' -th image shown in a lower right part of Fig. 37. 

The calculation circuit 24 converts positions of the 
green pixel G(jg - 1) (ig - 1) of the k'-th image and the red 

20 pixel R(jr - 1) (ir - 1) of the k 1 1 -th image into positions 
in the coordinate system of the first image on the upper 
side of Fig. 37 by using conversion parameters (6k 1 , Tk'x, 
Tk'y, Sk') and Ok » ' , Tk !, x, Tk''y, Sk ' ' ) in a similar 
manner to that according to the third embodiment. 

25 The calculation circuit 24 then calculates the distance 
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between the green pixel G(jg - 1) (ig - 1) of the k'-th image 
at the position converted into the coordinate system of the 
first image and the red pixel R(jr - l)(ir - 1) of the k' 
th image at the position converted into the coordinate 
5 system of the first image. Furthermore, the calculation 
circuit 24 determines whether these two pixels can be 
regarded as locating at the same position by determining 
whether the calculated distance is less than a maximum 
allowable value (threshold value) delta (set to, for example, 

10 0.25 times the pixel size) . 

When positions of pixels G(jg - 1) (ig - 1) and R(jr - 
1) (ir - 1) are respectively given as positions (ig, jg) and 
(ir, jr) , if the position (ig, jg) of the pixel G(jg - 1) (ig 
- 1) in the coordinate system of the k'-th image is denoted 

15 as G( C ), the position (ir, jr) of the pixel R(jg - 1) (ig - 1) 
in the coordinate system of the k !, -th image is denoted as 
R( C ), the position (ig, jg) of the pixel G(jg - 1) (ig - 1) in 
the coordinate system of the first image is denoted as G C ( k .), 
and the position (ir, jr) of the pixel R(jg - 1) (ig - 1) in 

20 the coordinate system of the first image is denoted as R C (k')/ 
then a constraint requiring that the distance between point 
G c(k .) and point R c <k") should be within the allowable value 
delta is given by formula (37) shown below. 
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Dis[G c(lo ,R c(n ] 



= Dis 
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/Tk"x 
\Tk"y 



Sk 




cos (0k") -sin(0 k M )V ir 
sin(0k") cos(0k w )Ajr 



+ 



< delta 



• ■ ■ (37) 



Formula (37) is referred to as a distance constraint 
formula. In formula (23), Dis[G c(k .), R C (k">] denotes the 
distance between point G c(k .) and point R c <k")« The positions 
of point G c <k')) and point R C (k") are obtained by performing 
the affine transformation on positions (ig, jg) and (ir, jr) 
using conversion parameters (9k' , Tk'x, Tk'y, Sk ' ) and (8k !, # 
Tk' 'x, Tk* »y, Sk ■ ' ) . 

The calculation circuit 24 determines whether there are. 
a green pixel G(jg - l)(ig - 1) of the k'-th image and a red 
pixel; R(jr - 1) (ir - 1) of the k 1 ' -th image at positions 
that can be regarded as being the same within an allowable 
margin delta in a small region (I 1 ± dX, J ' ± dY) centered 
at a position (I 1 , J') in the coordinate system of the first 
image, that is, in a rectangular region whose corners are at 
(I' - dX, J 1 - dY) , (I" - dX, J ' + dY) , (I 1 + dX, J ' - dY) , 
and (I r + dX, J ! + dY) , wherein dX and dY are values that 
define the region in the vicinity of the position (x, y) and 
they are set to be equal to, for example, 2 times the size 
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of a pixel in the X direction and Y direction, respectively. 

In other words, the calculation circuit 24 determines 
(k 1 , ig, jg) and (k' ! , ir, jr) that satisfy formula (37) 
within the small region (I 1 ± dX, J ' ± dY) centered at the 
5 position J') of interest in the coordinate system of 

the first image, that is, in the rectangular region whose 
corners are at (I' - dX, J' - dY) , (I' - dX, J 1 + dY) , (I' + 
dX, J' - dY) , and (I- + dX, J' + dY) . 

The calculation circuit 24 then acquires pixel values 
10 Gobs(k , / ig, jg) and Robs(k ,, / ir, jr) respectively 

corresponding to (k*, ig, jg) and (k fl , ir, jr) determined. 

The calculation circuit 24 determines (k', ig, jg) and 
(k' 1 , ir, jr) that satisfy formula (37) for all possible 
combinations of k' and k'' within the range form 1 to N. 
15 In general, a plurality of combinations of (k', ig, jg) 

and (k 1 ', ir, jr) is detected, and the calculation circuit 
24 plots pixel values Gobs(k', ig, jg) and Robs (k 11 , ir, jr) 
corresponding to the detected (k 1 , ig, jg) and (k' 1 , ir, jr) 
in the GR space in which the horizontal axis represents the 
20 G signal (Gobs(k', ig, jg) ) and the vertical axis represents 
the R signal (Robs(k'', ir, jr)), as shown in Fig. 38. 

Fig. 3 8 shows the GR space in which sets of pixel 
values Gobs(k', ig, jg) and Robs (k 11 , ir, jr) satisfying 
formula (37) are plotted. 
25 In Fig. 38, each x-like mark indicates a pair of pixel 
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values Gobs(k', ig, jg) and Robs (k' ' , ir, jr) corresponding 
to (k', ig, jg) and (k 11 , ir, jr) detected by the 
calculation circuit 24, that is, a pair of pixel values 
Gobs(k', ig, jg) and Robs(k'*, ir, jr) satisfying formula 
5 (37) . 

The plots shown in Fig. 38 indicate that there is a 
correlation between the true green light intensity Lg(I', 
J ! ) and the true red light intensity Lr ( I ' , J') to be 
determined in the vicinity of the position (I 1 , J') of 
10 interest. 

Thus, in the seventh embodiment, in addition to spring 
constraints given by equations (25) to (27) in the third 
embodiment, the correlation between green and red colors 
shown in Fig. 3 8 is added as a constrain. 

15 The calculation circuit 24 performs principal component 

analysis for the plurality of points plotted in the GR space 
shown in Fig. 38, wherein each plotted point corresponds to 
a pair of pixel values Gobs(k', ig, jg) and Robs(k ,, / ir, 
jr) that the calculation circuit 24 has detected as 

20 satisfying the distance constraint (37) . 

The calculation circuit 24 then determines a principal 
direction as a result of the principal component analysis 
and determines a line Qg x G + Qr x R = Qc representing the 
determined direction (axis) where Qg, Qr, and Qc are 

25 constants associated with the line in the GR space, and G 
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and R are variables associated with the G and R signals in 
the GR space. Furthermore, the calculation circuit 24 
determines the variance in a direction perpendicular to the 
principal direction . 
5 Herein, a spring with a natural length of 0 and with a 

spring constant (strength) H is introduced. The spring 
constant may be an arbitrary monotonically decreasing 
function of the variance of the component perpendicular to 
the direction of the principal component determined in the 

10 GR space. For example, H = (1 / variance) may be used. In 
any case, the strength of the spring (spring constant) 
increases with decreasing variance. 

One end of the spring with the spring constant H is 
connected to point (Lg(I', J 1 ) , Lr ( I ' , J')) in the GR space, 

15 and the other end of the spring is connected such that it 

can movable to an arbitrary point on the line Qg x G + Qr x 
R = Qc. As a result, the point (Lg(I», J'), Lr(I', J')) is 
pulled in a direction of the perpendicular line from the 
point (Lg(I', J'), Lr ( I 1 , J')) to the line Qg x G + Qr x R = 

20 Qc. That is, the point (Lg(I', J'), Lr(I', J 1 ) ) is pulled 
such that the color correlation constraint (between the G 
and R signals) is satisfied. 

Herein, if the distance between the point (Lgd 1 , J T ), 
Lr(I*, J')) and the line Qg x G + Qr x R = Qc extending in 

2 5 the direction of the main component is represented by u, 
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then the pulling force of the spring with the spring 
constant H exerted on the point (Lg(I', J'), Lr ( I 1 , J')) is 
given by H x u (hereinafter, this force will be referred to 
as GR spring force) . The GR spring force H x u in the GR 
5 space is broken down into a G signal component and a R 
signal component, and these components are respectively- 
added to the left-hand sides of equations (25) and (26) such 
that the color correlation is taken into account in 
equations (25) and (26) . 

10 The color correlation between the green (G) signal and 

the B (Blue) signal is dealt with as follows. 

As in the case of the color correlation between the 
green and red signals described above with reference to Fig. 
37, the calculation circuit 24 converts positions of the 

15 green pixel G(jg - 1) (ig - 1) of the k ! -th image and the 
blue pixel B(jb - 1) (ib - 1) of the k !,, -th image into 
positions in the coordinate system of the first image by 
using conversion parameters (0k 1 , Tk'x, Tk'y, Sk 1 ) and 
Ok' 11 , Tk' ' 'x, Tk' ■ ' y, Sk ■ ■ ' ) . 

20 The calculation circuit 24 then calculates the distance 

between the green pixel G(jg - 1) (ig - 1) of the k'-th image 
at the position converted into the coordinate system of the 
first image and the blue pixel B(jb - 1) (ib - 1) of the 
k' ' * -th image at the position converted into the coordinate 

25 system of the first image. Furthermore, the calculation 
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circuit 24 determines whether these two pixels can be 
regarded as locating at the same position by determining 
whether the calculated distance is less than a maximum 
allowable value (threshold value) delta. 
5 When positions of pixels G(jg - 1) (ig - 1) and B(jb - 

1) (ib - 1) are respectively given as positions (ig, jg) and 
(ib, jb) , if the position (ig, jg) of the pixel G(jg - 1) (ig 
- 1) in the coordinate system of the k'-th image is denoted 
as point G (c) , the position (ib, jb) of the pixel B(jb - 

10 1) (ib - 1) in the coordinate system of the k' 1 ' -th image is 
denoted as point B (c)/ the position (ig, jg) of the pixel 
G(jg - 1) (ig - 1) in the coordinate system of the first 
image is denoted as G C( } C .), and the position (ib, jb) of the 
pixel B(jb - 1) (ib - 1) in the coordinate system of the 

15 first image is denoted as B c(k ...), then a constraint requiring 
that the distance between point G c(k .) and point B C ( k ...> should 
be within the allowable value delta is given by formula (38) 
shown below. 

Dis £ G c<k')> B c(k"')] 



= Dis 



,/cos(0k') -sin(0k')Vig\ [Tk'x 

Sk vsin(ek') coscen/Vjg/VTky 

„ /cos(0k'") -s\n(8W")\( ib\ flk m x 
\s i n ( 6 k'") cos ( 0 k"')A J b / + \ Tk '"y J. 



< delta 



20 • • • (38) 
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Formula (38) is referred to as a distance constraint 
formula. In formula (23), Dis[G C ( k .), B C (k"->] denotes the 
distance between point G c(k .) and point B c(k ...). The positions 
of point G c(k .)) and point B c(k ...) are obtained by performing 
5 the affine transformation on positions (ig, jg) and (ib, jb) 
using conversion parameters (0k 1 , Tk'x, Tk'y, Sk ' ) and 
(9k 1 '', Tk' ' 'x, Tk* ' 'y, Sk ■ ' ' ) . 

The calculation circuit 24 determines whether there are 
a green pixel G(jg - 1) (ig - 1) of the k ! -th image and a 

10 blue pixel B(jb - 1) (ib - 1) of the k«''-th image at 

positions that can be regarded as being the same within an 
allowable margin delta in a small region (I* ± dX, J' ± dY) 
centered at a position (I 1 , J') in the coordinate system of 
the first image, that is, in a rectangular region whose 

15 corners are at (I' - dX, J' - dY) , (I 1 - dX, J' + dY) , (I' + 
dX, J f - dY) , and (I 1 + dX, J ! + dY) , wherein dX and dY are 
values that define the region in the vicinity of the 
position (x, y) and they are set to be equal to, for example, 
2 times the size of a pixel in the X direction and Y 

20 direction, respectively. 

In other words, the calculation circuit 24 determines 
(k', ig, jg) and (k'', ib, jb) that satisfy formula (38) 
within the small region (I 1 ± dX, J' ± dY) centered at the 
position (I 1 , J') of interest in the coordinate system of 

25 the first image, that is, in the rectangular region whose 
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corners are at ( I 1 - dX, J ' - dY) , (I 1 - dX, J 1 + dY) , (I' + 
dX, J' - dY) , and (I 1 + dX, J' + dY) . 

Tthe calculation circuit 24 then acquires pixel values 
Gobs(k', ig, jg) and Bobs(k ,,, / ib, jb) respectively 
5 corresponding to (k', ig, jg) and (k' 1 ', ib, jb) determined. 

The calculation circuit 24 determines (k 1 , ig, jg) and 
(k''», ib, jb) that satisfy formula (38) for all possible 
combinations of k' and k' ' 1 within the range form 1 to N. 

In general, a plurality of combinations of (k', ig, jg) 
10 and (k 1 ' ' , ib, jb) is detected, and the calculation circuit 
24 plots pixel values Gobs(k', ig, jg) and BobsCk'' 1 , ib, 
jb) corresponding to the detected (k 1 , ig, jg) and (k' 1 1 , ib, 
jb) in the GB space in which the horizontal axis represents 
the G signal (Gobs(k ! , ig, jg) ) and the vertical axis 
15 represents the B signal (Bobs (k ! ' 1 , ib, jb) ) . 

Thus, in the seventh embodiment, in addition to spring 
constraints given by equations (25) to (27) in the third 
embodiment, the correlation between green and blue colors 
shown is added as a constrain. 
20 The calculation circuit 24 performs principal component 

analysis for the plurality of points plotted in the GB space, 
wherein each plotted point corresponds to a pair of pixel 
values Gobs(k f , ig, jg) and Bobs(k 11 ', ib, jb) that the 
calculation circuit 24 has detected as satisfying the 
25 distance constraint (38) . 
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The calculation circuit 24 then determines a principal 
direction as a result of the principal component analysis 
and determines a line Qg ' x G + Qb' x R = Qc ' representing 
the determined direction (axis), where Qg 1 , Qb ' , and Qc ' are 
5 constants associated with the line in the GB space, and G 
and B are variables associated with the G and B signals in 
the GB space. Furthermore, the calculation circuit 24 
determines the variance in a direction perpendicular to the 
principal direction . 

10 Herein, a spring with a natural length of 0 and with a 

spring constant (strength) H is introduced. The spring 
constant may be an arbitrary monotonically decreasing 
function of the variance of the component perpendicular to 
the direction of the principal component determined in the 

15 GB space. For example, H = (1 / variance) may be used. In 
any case, the strength of the spring (spring constant) 
increases with decreasing variance . 

One end of the spring with the spring constant H is 
connected to point (Lg(I', J'), Lb(I', J')) in the GB space, 

20 and the other end of the spring is connected such that it 

can movable to an arbitrary point on the line Qg 1 x G + Qb ' 
x B = Qc'. As a result, the point (Lg(I», J 1 ), Lb ( I 1 , J')) 
is pulled in a direction of the perpendicular line from the 
point (Lg(I», J'), Lb(I', J')) to the line Qg ' x G + Qb' x B 

25 = Qc'. That is, the point (Lg(I', J'), Lb (I 1 , J»)) is 
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pulled such that the color correlation constraint (between 
the G and B signals) is satisfied. 

Herein, if the distance between the point (Lgd 1 , J'), 
Lb(I', J')) and the line Qg 1 x G + Qb ' x B = Qc 1 extending 
5 in the direction of the main component is represented by u, 
then the pulling force of the spring with the spring 
constant H exerted on the point (Lg(I ! , J'), Lb(I' , J')) is 
given by H x u (hereinafter, this force will be referred to 
as GB spring force) . The GB spring force H x u in the GB 

10 space is broken down into a G signal component and a B 
signal component, and these components are respectively 
added to the left-hand sides of equations (25) and (27) such 
that the color correlation is taken into account in 
equations (25) and (27) . 

15 Although in the present embodiment, the constraint 

associated with the color correlation in the GR space and 
the constraint associated with the color correlation in the 
GB space, a similar constraint associated with a color 
correlation between the R signal and the B signal (in a RB 

2 0 space) may also be employed in addition to the above 
constraints . 

Now, referring to a flow chart shown in Fig. 39, a 
correct image estimation process in step S4 shown in Fig. 2 
according to a seventh embodiment of the present invention 
25 is described. 
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Steps S171 to S177 are similar to steps S71 to S77 in 
the correct image estimation process according to the third 
embodiment described above with reference to Fig. 24. 
That is, in step S171, the calculation circuit 24 
5 selects a position (I 1 , J') in the reference coordinate 
system (hereinafter, such a selected position will be 
referred to a position (I 1 , J') of interest). Note that the 
position (I 1 , J 1 ) of interest is at the center (i - 0.5, j - 
0.5) of a (i-th, j-th) pixel of the first image taken as the 

10 reference image. 

Thereafter, the process proceeds from step S171 to step 
S172 . In step S172, the calculation circuit 24 determines 
all combinations of integers (k, ig, jg) that satisfy I 1 - 1 
< x < I 1 +1 and J' - 1 < y < J 1 +1 with respect to the 

15 position (x, y) in the reference coordinate system obtained 
by converting, based on conversion parameters (0k, Tkx, Tky, 
Sk) , the center position (ig - 0.5, jg - 0.5) of the pixel 
that senses a green component of the k-th image, for the 
position (I 1 , J') of interest of each of first to N-th 

20 images. Thereafter, the process proceeds to step S173 . 

In step S173, the calculation circuit 24 produces a 
spring equation in the form (25) for green light intensity, 
using all combinations of (k, ig, jg) determined in step 
S172 . Thereafter, the process proceeds to step S174. 

25 In step S174, the calculation circuit 24 determines all 
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combinations of integers (k, ir, jr) that satisfy I' - 1 < x 
< I ' + 1 and J' -l<y<J'+l with respect to the 
position (x, y) in the reference coordinate system obtained 
by converting, based on conversion parameters (9k, Tkx, Tky, 
5 Sk) , the center position (ir - 0.5, jr - 0.5) of the pixel 
that senses a red component of the k-th image, for the 
position (I 1 , J') of interest of each of first to N-th 
images. Thereafter, the process proceeds to step S175. 

In step S175, the calculation circuit 24 produces a 

10 spring equation in the form (26) for red light intensity, 
using all combinations of (k, ir, jr) determined in step 
S174 . Thereafter, the process proceeds to step S176 . 

In step S176, the calculation circuit 24 determines all 
combinations of integers (k, ib, jb) that satisfy I' - 1 < x 

15 < 1 1 + 1 and J' - 1 < y < J ' +1 with respect to the 

position (x, y) in the reference coordinate system obtained 
by converting, based on conversion parameters (0k, Tkx, Tky, 
Sk) , the center position (ib - 0.5, jb - 0.5) of the pixel 
that senses a blue component of the k-th image, for the 

20 position (I 1 , J') of interest of each of first to N-th 
images. Thereafter, the process proceeds to step S177. 

In step S177, the calculation circuit 24 produces a 
spring equation in the form (27) for blue light intensity, 
using all combinations of (k, ib, jb) determined in step 

25 S176. Thereafter, the process proceeds to step S178. 
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In step S178, the calculation circuit 24 determines all 
pairs of (k', ig, jg) and (k ,! , ir, jr) that satisfy formula 
(37) in the region defined by (I 1 ± dx, J ! ± dY) and 
centered at the position (I 1 , J') of interest. Furthermore, 
5 the calculation circuit 24 plots, in the GR space, points 
(Gobs(k', ig, jg) and Robs(k'', ir, jr)) identified by the 
determined pairs of (k 1 , ig, jg) and (k' ir, jr), and 
performs principal component analysis. The calculation 
circuit 24 then determines the variance in a direction 

10 perpendicular to the principal direction and employs the 
reciprocal of the variance as the spring constant H. 
Furthermore, the calculation circuit 24 determines a line Qg 
x G + Qr x R = Qc representing the direction of the 
principal component and defines an unknown quantity H x u 

15 indicating the GR spring force where u is the distance 

between a point (Lg(I', J'), Lr(r , J 1 )) in the GR space and 
the line Qg x G + Qr x R = Qc . The calculation circuit 24 
adds the G signal component of the GR spring force H x u to 
the left-hand side of equation (25) and adds the R signal 

20 component of the GR spring force H x u to the left-hand side 
of equation (26) . Thereafter, the process proceeds from 
step S178 to step S179. 

In step S179, and the calculation circuit 24 determines 
all pairs of (k', ig, jg) and (k 111 , ib, jb) that satisfy 

25 formula (38) in the region defined by (I' ± dx, J» ± dY) and 
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centered at the position (I 1 , J ! ) of interest. Furthermore, 
the calculation circuit 24 plots, in the GB space, points 
(Gobs(k', ig, jg) and Bobs(k tlf , ib, jb) ) identified by the 
determined pairs of (k 1 , ig, jg) and (k' ' ' , ib, jb) , and 
5 performs principal component analysis. The calculation 
circuit 24 then determines the variance in a direction 
perpendicular to the principal direction and employs the 
reciprocal of the variance as the spring constant H. 
Furthermore, the calculation circuit 24 determines a line 

10 Qg' x G + Qb ' x R = Qc 1 representing the direction of the 
principal component and defines an unknown quantity H x u 
indicating the GB spring force where u is the distance 
between a point (Lg(I', J'), Lb (I 1 , J')) in the GB space and 
the line Qg 1 x G + Qb ' x R = Qc ' . The calculation circuit 

15 24 adds the G signal component of the GB spring force H x u 
to the left-hand side of equation (25) and adds the B signal 
component of the GB spring force H x u to the left-hand side 
of equation (27) . Thereafter, the process proceeds from 
step S179 to step S180. 

20 In step S180, the calculation circuit 24 solves linear 

equations, that is, the spring equation associated with the 
green light intensity and including the additional terms 
associated with the color correlations in the GR and GB 
spaces, the spring equation associated with the red light 

25 intensity and including the additional term associated with 
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the color correlation in the GR space, and the spring 
equation associated with the blue light intensity and 
including the additional term associated with the color 
correlation in the GB space thereby determining the true 
5 green light intensity Lg(I', J'), the true red light 

intensity Lr(r , J'), and the true blue light intensity 
Lb(I', J') at the position J') of interest. Thereafter, 

the process proceeds to step S181. 

In step S181, the calculation circuit 24 determines 

10 whether the above -described process is completed for all 

possible positions (I', J 1 ), that is, whether the true green 
light intensity Lg(I', J'), the true red light intensity 
Lr(I', J'), and the true blue light intensity Lb ( I 1 , J') 
have been determined by taking the center position of each 

15 of all pixels of the first image as the position (I', J') of 
interest . 

If it is determined in step S181 that the process is 
not completed for all possible positions (I', J') of 
interest, the process returns to step S171 to repeat steps 

20 S171 to S181. That is, the calculation circuit 24 selects, 
as a new position of interest, a position (I 1 , J ! ) that has 
not yet been selected as the position of interest, and 
determines the true green light intensity Lg ( I ' , J')/ the 
true red light intensity Lr(I', J 1 ), and the true blue light 

25 intensity Lb (I 1 , J') at the position (I 1 , J') of interest. 
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On the other hand in the case in which it is determined 
in step S181 that the process is completed by taking all 
possible position (I 1 , J 1 ) as the position of interest, the 
process proceeds to step S182 . In step S182, the 
5 calculation circuit 24 estimates a correct image (signal) 

from the true green light intensity Lg(I', J'), the true red 
light intensity Lr(I', J 1 ) , and the true blue light 
intensity Lb(I', J') determined in step S180. The resultant 
image is supplied as an output image to the D/A converter 9 

10 or the codec 12, and the current process is exited. For 

example, for the (i-th, j-th) pixel, the calculation circuit 
24 estimates the correct image signal thereof from the true 
green light intensity Lg(i - 0.5, j - 0.5) determined as the 
green value (G signal) in step S180, the true red light 

15 intensity Lr(i - 0.5, j - 0.5) determined as the red value 
(R signal) in step S180, and the true blue light intensity 
Lb(i - 0.5, j - 0.5) determined as the blue value (B signal) 
in step S180. The calculation circuit 24 estimates the 
output image by estimating the image signal for all pixels 

20 whose center is located at the selected position (I', J'). 

As described above, in the seventh embodiment, the 
error due to noise or the like is reduced by taking into 
account the color correlation when the output image is 
produced. This makes it possible to produce a sharper 

25 output image .more likely to be the correct image. 
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To provide a better understanding of the features of 
the present invention, the difference between the present 
invention and the conventional technique is further 
described below. 

5 Fig. 4 0 shows a processing flow of the method according 

to the present invention and that of the conventional method. 

As shown in Fig. 40, many methods (denoted by (A) in 
Fig. 40) to convert an image output from a single-plane 
sensor (for example, a Bayer-array image in which each pixel 

10 has only one of R, G, and B signals) to a normal color image 
(having R, G, and B signals for each pixel) are known in the 
art. In fact, this conversion capability is implemented in 
some digital still cameras using a single-plane sensor 
available in the market. This conversion is called 

15 demosaicing. 

A method (denoted by (B) in Fig. 40) of producing a 
sharp color image (having R, G, and B signals for each 
pixel) (with a high signal-to-noise ratio) from a plurality 
of dark color images (having R, G, and B signals for each 

20 pixel) (with a low signal-to-noise ratio) is also known in 
the art. Specific examples of this method include Japanese 
Unexamined Patent Application Publication No. 2000-217032 
and Japanese Unexamined Patent Application Publication No. 
2000-224460 cited earlier in Background Art. 

25 By combining these two methods, it is possible to 
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produce a sharp color image (having R, G, and B signals for 
each pixel) (with a high signal-to-noise ratio) from an 
image output from a single -plane sensor (for example, a 
Bayer-array image in which each pixel has only one of R, G, 
and B signals) . 

More specifically, a demosaicing process 202-m denoted 
by (A) in Fig. 40 is performed on each of a plurality (m) of 
dark images 201-m output from a single-plane sensor (for 
example, Bayer- array images in which each pixel has only one 
of R, G, and B signals) (with a low signal-to-noise ratio) 
to produce dark color images (having R, G, and B signals for 
each pixel) (with a low signal-to-noise ratio) 203-m. 
Thereafter, a sharp color image (having R, G, and B signals 
for each pixel) (with a high signal-to-noise ratio) 204 is 
produced from the color images 203 -m by the method denoted 
by (B) in Fig. 40 such as a method disclosed in Japanese 
Unexamined Patent Application Publication No. 2000-217032 or 
Japanese Unexamined Patent Application Publication No. 2000- 
224460. Note that m is an integer indicating the number of 
images . 

Interpolation or similar processing performed in the 
demosacing process causes a reduction in sharpness of the 
image. In the process disclosed in Japanese Unexamined 
Patent Application Publication No. 2000-217032 or 2000- 
224460, interpolation is performed for registration among 
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digitally sampled data, and thus degradation in sharpness 
also occurs. That is, interpolation is performed in the 
process denoted by (A) in Fig. 40 and also in the process 
(B) . The interpolation performed a total of two times 
5 causes significant degradation in sharpness. The 

interpolation is used to reproduce missing data at a 
particular point from data at adjacent points. The more 
times interpolation is performed, the more degradation in 
sharpness occurs . 

10 In contrast, in the present invention, a sharp color 

image (having R, G, and B signals for each pixel) (with a 
high signal-to-noise ratio) is produced directly from an 
image output from a single-plane sensor (for example, a 
Bayer-array image in which each pixel has only one of R, G, 

15 and B signals) . Therefore, interpolation or similar 

processing is performed only once. Thus, it is possible to 
produce a sharper image than the conventional method can. 
As can be seen from the above discussion, it is obvious that 
the present invention is different from the conventional 

20 technique and the present invention has great advantages. 

Another problem with the conventional technique is that 
demosacing is performed for each image. For example, when 
eight images are given as the plurality of images, 
demosacing (denoted by (A) in Fig. 40) is performed eight 

25 times. This implies that a large amount of computation is 
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required. 

The different between the present invention and the 
conventional technique is further described below. 

In the conventional technique disclosed in Japanese 
5 Unexamined Patent Application Publication No. 2000-217032 
cited earlier in Background Art, when a plurality of images 
are superimposed, images are displaced to register them such 
that a positional displacement among original images due to 
a camera shake is cancelled out. To produce images 

10 displaced by an amount necessary to cancel out the camera 
shake, interpolation is necessary. Superimposing N images 
together is performed by superimposing N data at each pixel 
of an output image. After N images are superimposed, the 
pixel value of each pixel is divided by a particular value 

15 (equal to N or equal to (optimum exposure time) / (actual 

exposure time) / N) . It is also known in the art to achieve 
a high resolution image by capturing a plurality of images 
while displacing pixel positions by a fixed amount each time 
one image is captured and superimposing the plurality of 

20 captured images while properly registering them. In this 
technique, the amount of displacement is fixed, the pixel 
value of each pixel of the output image is divided 
(normalized) by a fixed value. 

In contrast, in the first or second embodiment 

25 according to the present invention, the number of pixel 
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values observed at a given position (x, y) of an output 
image available to estimate a true value varies depending on 
a situation. For example, when a picture of a scene is 
taken by a user, there is a possibility that there are a 
5 large number of input pixel values Gobs(k, ig, jg) in the 
vicinity of a position (x, y) . In such a case, a large 
number of constraints associated with the position (x, y) in 
the form of a mathematical expression (20) are available. 
Conversely, if there are only a small number of input pixels 

10 Gobs(k, ig, jg) in the vicinity of the position (x, y) , only 
a small number of constraints associated with the position 
(x, y) in the form of the mathematical expression (20) are 
available. As described above, the number of mathematical 
constraints associated with each position (x, y) of an 

15 output image is variable depending on the state in which a 
picture is taken (more particularly, for example, depending 
on the camera shake) . That is, in the process of estimating 
the true value at each pixel, observation points of 
candidates for pixel values to be used in the estimation and 

2 0 the number thereof are adapt ively changed depending on the 

state in which the picture is taken (depending on the motion 
among the plurality of captured images) . For example, the 
number of observation points of candidates for pixel values, 
and a particular number of observations are selected from 

25 the candidates. Alternatively, both the number of 
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observation points of candidates and the number of 
observation points to be selected may be changed. The above - 
described two methods may be selectively used depending on 
the positions, numbers, and/or distribution of observation 
5 points. 

In the third to seventh embodiments of the present 
invention, when images are superimposed, images are 
displaced so as to cancel out displacement due to a camera 
shake. However, in the Bayer-array structure, there is not 

10 necessarily data to be added, at all output positions. When 
a k-th image is displaced in position, if there is no 
position-corrected data within a 2 x 2 pixel area shown in 
Fig. 21, any data of the k-th image has no contribution to 
those 2x2 pixels of the output image. Conversely, if 

15 there are two position-corrected data within the 2x2 pixel 
area shown in Fig. 21, the two (weighted) data are added to 
the output pixel. For example, if ten data are added at a 
particular pixel, the final pixel value of the output image 
is obtained by dividing the sum by ten. If 3.5 data are 

20 added together at another pixel, the pixel value at that 

pixel of the output image is obtained by dividing the sum by 
3.5. That is, in the present invention, the number of times 
summation £ is executed in equations (25) , (26) , and (27) 
varies depending on the state in which a picture is taken 

25 (more particularly, for example, depending on the camera 
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shake) . That is, in the process of estimating the true 
value at each pixel, observation points of candidates for 
pixel values to be used in the estimation and the number 
thereof are adaptively changed depending on the state in 
5 which the picture is taken (depending on the motion among 
the plurality of captured images) . 

In the present invention, as described above, when a 
sharp image is restored from a plurality of images output 
from a single-plane sensor (Bayer-array sensor) , the number 

10 of constraint equations or the weights are varied depending 
on the situation. In this regard, the present invention is 
very different from the conventional techniques. 

The sequence of processing steps described above may be 
performed by means of hardware or software. For example, 

15 the digital camera 1 may be realized by executing a program 
on a computer such as that shown in Fig. 41. 

In Fig. 41, a CPU (Central Processing Unit) 301 
performs various processes in accordance with a program 
stored in a ROM (Read Only Memory) 3 02 or in accordance with 

20 a program loaded into a RAM (Random Access Memory) 303 from 
a storage unit 308. The RAM 303 is also used to store data 
used by the CPU 301 in the execution of various processes. 
The CPU 301 executes the process performed by the motion 
detector 23 and the calculation circuit 24 of the signal 

25 processor 7. 
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The CPU 3 01, the ROM 3 02, and the RAM 3 03 are connected 
to each other via a bus 304. The bus 304 is also connected 
to an input/output interface 305. 

The input /output interface 3 05 is connected to an input 
5 unit 3 06 including a keyboard, a mouse, and/or the like, an 
output unit 307 including a display such as a CRT (Cathode 
Ray Tube) or a LCD (Liquid Crystal Display) , a loudspeaker, 
and/or the like, the storage unit 3 08 such as a hard disk, 
and a communication unit 309 including a modem, a terminal 

10 adapter, and/or the like. The communication unit 309 allows 
communication via a network such as the Internet. The 
imaging unit 311 includes the imaging device 4 shown in Fig. 
1 and serves to capture images of a subject. The resultant 
image data of the subject is supplied to the CPU 3 01 via the 

15 input/output interface 305. 

The input /output interface 3 05 is also connected to a 
drive 310, as required. A storage medium such as a magnetic 
disk 321, an optical disk 322, a magnetooptical disk 323, or 
a semiconductor memory 324 is mounted on the drive 310 as 

20 required, and a computer program is read from the storage 

medium and installed into the storage unit 308, as required. 

Now, a correct image estimation process performed by 
the signal processor 7 according to an eighth embodiment of 
the present invention is described. In this eighth 

25 embodiment, the third embodiment described above with 
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reference to Fig. 24 is partially improved. 

In the third embodiment described above, data (light 
intensity) sensed by each pixel of the imaging device 4 (Fig. 
1) is regarded as point -sampled data, the relationship 
5 between a pixel value observed at the center of each pixel 
and an ideal image signal of a sharp image including no blur 
due to a camera shake is represented by the spring model, 
and a sharp image likely to be a true image is determined 
based on the spring model. 

10 For example, of G, R, and B signals of an image signal 

of a sharp image with no blur due to camera shake, 
estimation of the true G signal according to the third 
embodiment is performed as follows. That is, the constraint 
on the true green light intensity Lg ( I ' , J') to be 

15 determined for the center position (I r , J') of each pixel is 
expressed by the spring equation (25) associated with the 
green light intensity, as described earlier. 

In equation (25), {^2 - F((x, y), (I', J'))} of the 
spring constant {^2 - F((x, y), (I', J'))}/(Mk x E) 

20 represents the weight that decreases with distance between 
the position (I', J') and the position (x, y) in the 
vicinity of the position (I', J'). This means that the 
pulling force of the spring exerted between the pixel value 
Gobs(k, ig, jg) at the position (x, y) and the true green 

25 light intensity Lg(I f , J 1 ) at the position (I 1 , J') 
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decreases with distance between the position (I', J') and 
the position (x, y) . Conversely, the pulling force of the 
spring increases with decreasing distance between the 
position (I*, J') and the position (x, y) . 
5 On the other hand, (Mk x E) of the spring constant {^2 - 

F((x, y) , (I 1 , J'))}/(Mk x E) in equation (25) represents 
residual noise E included that cannot be removed by the 
correlated double sampler 5 (Fig. 1) gained up by a factor 
of Mk (k = 1 to N) . Thus, 1/ (Mk x E) represents a weight 

10 associated with noise, which decreases with noise E. 

Therefore, the pulling force of the spring exerted between 
the pixel value Gobs (k, ig, jg) at the position (x, y) and 
the true green light intensity Lg ( I 1 , J') at the position 
(I 1 , J') decreases with increasing noise E and increases 

15 with decreasing noise. 

In the eighth embodiment, {^2 - F((x, y) , (I', J'))} in 
equation (25) , that is, the weight depending on the distance 
between the position (I 1 , J') and the position (x, y) of the 
pixel value Gobs(k, ig, jg) (hereinafter, referred to simply 

20 as a distance weight) is replaced with a function including 
cubic functions: Cubic (I ' - x) x Cubic (J 1 - y) . That is, in 
the eighth embodiment, {V2 - F((x, y) , (I', J'))} in 
equation (25) is replaced with Cubic (I 1 - x) x Cubic (J' - y) . 
The cubic function Cubic (z) is in a form (39) shown 

25 below. 
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'(a+2)|z| 3 -(a+3)|z| 2 +1 (|z|<1) 
Cubic(z) = \ a|z| 3 -5a|z| 2 +8a|z|-4a (1<|z|<2) 
^0 (2<|z|) 



• • • (39) 



where a is a predetermined constant, which is set, for 
example, to -1. 

As shown in Fig. 42, the cubic function Cubic (z) has a 
5 value of 0 when 2 < |z| or when |z| =1, and the cubic 

function Cubic (z) is negative when 1< |z| < 2. When |z| < 1, 
Cubic (z) is positive and decreases with |z|. In other words, 
the cubic function Cubic (z) serves as a lowpass filter in 
the frequency domain (for Fourier- transformed variable) . 

10 In equation (25), if p-th pixel value Gobs(k, ig, jg) 

subjected to summation £ is denoted by v p , the spring 
constant {V2 - F((x, y), (I 1 , J'))}/(Mk x E) associated with 
the p-th pixel value v p is denoted as a spring constant K p , 
and if the resultant equation is solved with respect to the 

15 true green light intensity Lg(I', J'), equation (40) is 
obtained . 



where £ denotes the sum with respect to p. 

From equation (40), it can be seen that, in the spring 
20 model, the true green light intensity Lg(I', J') is given by 



Lg(I\ J') = 



IK P V P 
IK P 



(40) 



- 220 - 

S04P0831 

the weighted sum of pixel values v p wherein the spring 
constants K p are employed as weights. The true red light 
intensity Lr(I ! , J') and the true blue light intensity L.b(I', 
J') can be determined in a similar manner. 
5 In Fig. 43, for the position (I*, J'), when positions 

(ig - 0.5, jg - 0.5) of N captured images are converted into 
positions (x, y) in the reference coordinate system 
(coordinate system defined on the first image) , all 
combinations of integers k, ig, and jg that satisfy I 1 - 2 < 

10 x < I 1 + 2 and J ' - 2 < y < J' +2 are determined, and 

points Gil to G19 at which pixel values Gobs(k, ig, jg) are 
observed are shown . 

In Fig. 43, for a position (I 1 , J 1 ) denoted by a solid 
circle, combinations of integers (k, ig, jg) identifying 

15 nine pixel values Gobs(k, ig, jg) observed at points Gil to 
G19 denoted by open circles are obtained. Note that 
combinations of integers k, ig and jg are different among 
points Gil to G19. 

In the third embodiment described earlier, as shown in 

20 Fig. 21, for a position (I', J'), positions (ig - 0.5, jg - 
0.5) on N captured images are converted into positions (x, 
y) in the reference coordinate system (coordinate system 
defined on the first image) and all combinations of 
integers k, ig, and jg that satisfy I' - 1 < x < I ! +1 and 

25 J ' - 1 < y < J 1 +1 are determined. In contrast, in the 
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eighth embodiment, as shown in Fig. 43, combinations of 
integers k, ig, and jg that satisfy I' -2<x<I ! +2 and 
J ' - 2 < y < J' +2 are determined. As described above, the 
cubic function Cubic (z) in the form (39) has a value 
5 depending on argument z for - 2 < z < 2 (Cubic (z) has a 

value of 0 for any z in the range of |z| > 2) , and the true 
green light intensity Lg ( I ' , J") at the position ( 1 1 , J') is 
estimated using pixel values Gobs(k, ig, jg) observed in the 
range - 2 < z < 2, that is, in the range defined by I' - 2 < 

10 x < I' + 2 and J' - 2 < y< J ' +2. In the third embodiment 
described earlier, a position (ig - 0.5, jg - 0.5) on the k- 
th captured image is converted to a position ((ig - 0.5) (k) , 
(jg - 0.5) <k)) in the reference coordinate system defined on 
the first captured image as follows. That is, the motion 

15 detector 23- (k - 1) determines the af f ine conversion 

parameters (0k, Tkx, Tky, Sk) including a rotation angle 9k, 
a scale Sk, and a parallel displacement (Tkx, Tky) from the 
positional relationship between the first captured image and 
the k-th captured image. The resultant conversion 

20 parameters (9k, Tkx, Tky, Sk) are supplied to the 

calculation circuit 24 . Based on the conversion parameters 
(9k, Tkx, Tky, Sk) from the motion detector 23- (k - 1), the 
calculation circuit 24 converts the position (ig - 0.5, jg - 
0.5) on the k-th captured image to a position ( (ig - 0.5) 

25 (jg - 0.5) (k) ) in the reference coordinate system in 
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accordance with affine conversion equation (3) . 

On the other hand, in the eighth embodiment, the motion 
detector 23- (k - 1) determines affine conversion parameters 
of an affine conversion equation (41) representing the 
positional relationship between the first captured image and 
the k-th captured image. 



(k) 



X1 

^ Y1 (k>; 



ak' bk 
ck' dk' 



+ 

J V 



Xk 
Yk 



Tky' 



(41) 



More specifically, the motion detector 23- (k - 1) 
determines a matrix (ak ! , bk ' , ck T , dk 1 ) and a 2 -dimensional 

10 vector (Tkx 1 , Tky 1 ) in equation (41) representing the 

positional relationship between the first captured image and 
the k-th captured image, and supplies the resultant matrix 
(ak', bk ' , ck 1 , dk ' ) and the 2 -dimensional vector (Tkx', 
Tky 1 ) to the calculation circuit 24. Hereinafter, the 

15 affine conversion parameters including the matrix (ak', bk ■ , 
ck', dk ! ) and the 2 -dimensional vector (Tkx*, Tky 1 ) in 
equation (41) will be referred to simply as conversion 
parameters (ak 1 , bk 1 , dk 1 , Tkx', Tky'). 

If respective parameters are given such that ak ' = dk ' = 

20 Sk x cos (9k) and -bk» = ck 1 = Sk x sin (0k), equation (41) 
becomes equivalent to equation (3) . 

In equation (41) , as in equation (3) , (Xk, Yk) 
represents the position of a pixel of the k-th captured 
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image, and (Xl( k)/ Yl( k} ) represents a position on the first 
captured image (in the reference coordinate system) obtained 
by af fine-converting the position (Xk, Yk) in accordance 
with equation (41) . Note that a subscript (k) is used to 
indicate a position of the k-th captured image converted 
into the reference coordinate system. If the conversion 
parameters for the first captured image are given as (al', 
bl f , cl 1 , dl', Tlx', Tly') = (1, 0, 0, 1, 0, 0), then 
equation (41) holds for all values of k in the range from 1 
to N. 

For example, the motion detector 23-1 determines affine 
conversion parameters (a2', b2 ' , c2 ' , d2 1 , T2x f , T2y ! )of 
equation (42) representing the positional relationship 
between 'the first captured image and a second captured image, 
and supplies the resultant conversion parameters (a2 ' , b2 ' , 
c2 ' , d2 1 , T2x ! , T2y') to the calculation circuit 24. 



X1 (2)^ 




'a2' 


b2 J> 


fX2^ 


fT2x' > 


(2)^ 




,c2' 


d2', 


Y2 


h [T2y', 



On the other hand, the motion detector 23-2 determines 
affine conversion parameters (a3 ' , b3 ' , c3 ' , d3 ' , T3x' , 
T3y*)of equation (43) representing the positional 
relationship between the first captured image and a third 
captured image, and supplies the resultant conversion 
parameters (a3 ' , b3 ' , c3 ■ , d3 ' , T3x', T3y' ) to the 
calculation circuit 24 . 
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X1 <3) > 




'a3' 


b3 J> 


"X3" 


+ 




(3), 




,c3' 


d3' j 


,Y3, 







• • • (43) 



Equation (43) represents that a position (X3, Y3 ) on 
the third captured image is displaced, because of camera 
shake or the like, by an amount corresponding to the 
5 conversion parameters (a3 ' , b3 ' , c3 1 , d3 1 , T3x', T3y') with 
respect to a position (XI, Yl) on the first captured image. 

Instead of determining the conversion parameters (ak', 
bk', ck', dk', Tkx 1 , Tky') from the positional relationship 
of the k-th captured image with respect to the first 

10 captured image in the above-described manner, the conversion 
parameters (ak 1 , bk ' , ck ' , dk 1 , Tkx » , Tky' ) may be 
mechanically determined from a signal output from an 
acceleration sensor disposed on the digital camera 1. In 
the case of camera shaking that occurs when a camera is held 

15 in hands, the positional relationship between the first 
captured image and the k-th captured image includes 
substantially no rotation component. Therefore, the 
rotation component in the conversion parameters (ak f , bk 1 , 
ck*, dk' , Tkx', Tky 1 ) may be neglected (that is, it may be 

20 assumed that no rotation occurs) . 

In the eighth embodiment, the weight associated with 
noise in equation (25) is set based on the assumption that 
all N captured images are gained up by the same factor (by 
means of n'-bit shifting). That is, in the eight embodiment, 
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it is assumed that all N images (input images) captured by 
the imaging device 4 have brightness that is 1/M = 1/Mk 
times the brightness obtained with optimum exposure. For 
example, if captured images have brightness equal to 1/8 
5 times the brightness of an image captured with optimum 

exposure, then M = 8 , and thus shifter 21 (Fig. 1) gains up 
the captured image by performing 3 -bit shifting. 

In the eighth embodiment, because the shifter 21 gains 
up all N captured images by the same factor, the weight 
10 1/ (Mk x E) associated with noise in equation (25) can be 
simply rewritten as 1/ (M x E) . 

Thus, in this eighth embodiment, equation (2 5) used in 
the third embodiment can be rewritten as equation (44) . 



For the given position (I', J 1 ), the sum £ in equation 
(44) is taken for all combinations (k, ig, jg) that satisfy 
I 1 - 2 < x < I • + 2 and J 1 - 2 < y < J' + 2 with respect to 
the position (x, y) in the reference coordinate system 
20 obtained by converting, based on conversion parameters (ak', 
bk', ck', dk 1 , Tkx f , Tky'), the position (ig - 0.5, jg - 
0.5). In the specific example shown in Fig. 43, the sum is 
taken for nine combinations of integers (k, ig, jg) 



"Cubicd'-x) xCubicCJ'-y) 
MxE 

= 0 



x {Gobs(k, ig, jg)-Lg(I\ J')} 



15 



. . . (44) 
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corresponding to points Gil to G19. 

Equation (44) can be regarded as a spring equation 
associated with the green light intensity using the cubic 
function Cubic (z) according to the eighth embodiment, 
5 corresponding to the spring equation (25) associated with 
the green light intensity . 

In equation (44), the weight 1/ (M x E) associated with 
noise is constant regardless of the combinations of integers 
(k, ig, jg) subjected to the summation S, and thus the 
10 weight 1/ (M x E) can be moved out to the outside of E. Thus, 
equation (44) can be rewritten as (45) . 

1 



MxE 



Cubic(I'-x) xCubic(J'-y) x {Gobs(k, ig, jg) -Lg(I\ J')} 
= 0 

Cubic(I'-x) xCubic(J'-y) x {Gobs(k, ig, jg) -Lg(I\ J')} 
= 0 

• • • (45) 

If equation (45) is solved with respect to the true 
15 green light intensity Lg ( 1 1 , J'), equation (46) is obtained. 

ZCubic(I'-x) xfribic(J'-y) xGobs(k, ig, jg) 

= ZCubic(I'-x) xCubic(J'-y) x |_g(I\ J') 

, (J , n = Z Cubic(I'-x) xCubicCJ'-y) xGobs(k, ig, jg) 
gU ' ; ZCubicd'-x) xCubic(J'-y) 
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In this eighth embodiment, the true green light 
intensity Lg(I ! , J') is determined using equation (46). 

Equation (46) indicates that the green light intensity 
Lg(I ! , J 1 ) can be obtained by calculating the weighted sum 
of pixel values Gobs(k, ig, jg) by using Cubic (1 1 - x) x 
Cubic (J' - y) as the weight. Note that the weight Cubic (I' 
- x) x Cubic (J 1 - y) serves as a lowpass filter for the 
distance between the position (I 1 , J ! ) and the position (x, 
y) of the pixel value Gobs(k, ig, jg) . 

Hereinafter, equation (46) will be referred to as a 
weighted sum equation associated with the green light 
intensity. The denominator and the numerator of equation 

(46) associated with the weighted sum of green light 
intensities are separately rewritten as equations (47) and 

(48) , respectively. 

Z Cubicd'-x) xCubic(J'-y) xGobs(k, ig, jg) ... (47) 
ICubic(I'-x) xCubic(J'-y) ... (48) 

If the original data (light intensity) in the state in 
which the pixel value Gobs(k, ig, jg) is not yet gained up 
by a factor of M by the shifter 21 is denoted as D GO bs(k, ig, jg) , 
then equation (46) can be written as (49) . 
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Lg(I\ J') = 



Z Cub i c ( V -x) x Cub i c ( J' -y ) x M x D Gobs(k> ig> jg) 
ZCubic(r-x) xCubic(J'-y) 



■ • • (49) 

The true red light intensity Lr ( I 1 , J') and the true 
blue light intensity Lb (I 1 , J 1 ) at the position (I', J') of 
5 interest can be respectively represented by equations (50) 
and (51) , which is similar to equation (46) . 



10 weighted sum equation associated with the red light 

intensity. For the given position (I', J'), the sum S in 
equation (50) is taken for all combinations (k, ir, jr) that 
satisfy I' -2<x<I» +2 and J' -2<y<J' + 2 with 
respect to the position (x, y) in the reference coordinate 

15 system obtained by converting, based on conversion 

parameters (ak', bk 1 , ck ' , dk ' , Tkx', Tky 1 ) , the position 
(ir - 0.5, jr - 0.5) . 



Lr(I\ J') = 



ICubicd'-x) x Cubic (J' -y) xRobs(k, ir, jr) 
XCubic(F-x) xCubic(J'-y) 



• • • (50) 



Hereinafter, equation (50) will be referred to as a 



Lb(I\ J') = 



Z Cubic(r-x) x Cubic (J' -y) xBobs(k, ib, jb) 
ZCubic(I'-x) xCubic(J'-y) 



• • • (51) 



20 



Hereinafter, equation (51) will be referred to as a 



weighted sum equation associated with the blue light 



intensity. For the given position (I 



J ' ) , the sum S in 
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equation (51) is taken for all combinations (k, ib, jb) that 
satisfy I 1 -2<x<I'+2 and J' - 2 < y < J' +2 with 
respect to the position (x, y) in the reference coordinate 
system obtained by converting, based on conversion 
5 parameters (ak' , bk 1 , ck 1 , dk ' , Tkx' , Tky' ) , the position 
(ib - 0.5, jb - 0.5). 

The denominator and the numerator of equation (50) 
associated with the weighted sum of green light intensities 
are separately rewritten as equations (52) and (53), 
10 respectively. 



lCubic(I'-x) xCubic(J'-y) xRobs(k, ir, jr) 



(52) 



ZCubic(I'-x) xCubic(J'-y) 



(53) 



The denominator and the numerator of equation (51) 
associated with the weighted sum of green light intensities 
15 are separately rewritten as equations (54) and (55), 
respectively . 

I Cubic (I'-x) x Cubic (J' -y) xBobs(k, ib, jb) ... (54) 
I Cubic(I'-x) xCubic(J'-y) . . . (55) 

By calculating the weighted sum associated with the 
20 green light intensity according to equation (46) , the 
weighted sum associated with the red light intensity 
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according to equation (50) , and the weighted sum associated 
with the blue light intensity according to equation (51) , 
the calculation circuit 24 determines the true green light 
intensity Lg(r , J'), the true red light intensity Lr(r , 
5 J'), and the true blue light intensity Lb(I', J'). The 

process performed by the calculation circuit 24 to determine 
those light intensities is referred to as a normal process 
to distinguish it from an exceptional process that will be 
described later. 

10 The weighted sum associated with the green light 

intensity given by equation (46) , the weighted sum 
associated with the red light intensity given by equation 
(50) , and the weighted sum associated with the blue light 
intensity given by equation (51) will be further discussed 

15 below. 

Equation (46) indicates that the weighted sum 
associated with the green light intensity is obtained by 
dividing the sum of products of the pixel value Gobs(k, ig, 
jg) at the position (x, y) and the distance weight Cubic (I 1 

20 - x) x Cubic (J' - y) (that is, the sum of pixel values 

Gobs(k, ig, jg) weighted with the distance weight Cubic (I 1 - 
x) x Cubic (J' - y) ) by the sum of the distance weights 
Cubic (I 1 - x) x Cubic (J' - y) , that is, by dividing the 
value given by equation (47) that is equal to the numerator 

25 of equation (46) by the value given by equation (48) that is 
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equal to denominator of equation (46) . 

If the value given by equation (4 8) that is equal to 
the denominator of equation (46) becomes equal to 0, 
equation (46) representing the true green light intensity 
5 Lg(I', J') becomes unstable (uncertain), and thus the result 
is unreliable. In other words, at a position (I 1 , J') at 
which the value given by equation (48) that is equal to the 
denominator of equation (46) becomes equal to 0, small noise 
(error) included in the pixel values Gobs(k, ig, jg) in 

10 equation (46) that is equal to the numerator of equation 

(48) is divided by the denominator equal to 0, and thus the 
result becomes very large. In this case, the true green 
light intensity Lg ( I 1 , J') calculated according to equation 
(46) includes large noise and is unreliable. 

15 The value given by equation (48) that is equal to the 

denominator of equation (4 6) becomes equal to 0, for example, 
when at least one of cubic functions Cubic (I 1 - x) and 
Cubic (J 1 - y) becomes equal to 0 over all the range of the 
summation in equation (48) . As can be seen from Fig. 42, 

20 cubic functions Cubic (I* - x) or Cubic (J' - y) becomes equal 
to 0 when I 1 - x = ±1 or J' - y = ±1, that is, when x = I' ± 
1 or y = J' ±1. 

Therefore, for a given position (I 1 , J'), when 
positions (ig - 0.5, jg - 0.5) are converted into the 

25 reference coordinate system using conversion parameters (ak 1 , 
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bk', ck', dk', Tkx 1 , Tky'), if all positions (x, y) of pixel 
values Gobs(k, ig, jg) of (k, ig, jg) appearing within the 
range defined by I' - 2 < x < I 1 +2 and J' - 2 < y < J 1 +2 
satisfy x = I" ±1 or y = J' ±1, the denominator of 
5 equation (46) representing the true green light intensity 
Lg ( I' , J 1 ) becomes equal to 0 (or nearly equal to 0), and 
thus the result becomes unstable and unreliable. However, 
all positions (x, y) of pixel values Gobs(k, ig, jg) 
appearing within the range defined by 1 1 - 2 < x < I ' + 2 

10 and J' - 2 < y < J' +2 satisfy x=I l ±lory=J l ±l 
only in a very rare and exceptional state. Hereinafter, 
such a special state will be referred to simply as an 
exceptional state . 

Hereinafter, a position (x, y) converted from a 

15 position (k, i, j) on a k-th captured image into the 

reference coordinate system using conversion parameters (ak', 
bk ' , ck' , dk' , Tkx 1 , Tky' ) will be referred simply as a 
converted (corrected) position (x, y) . 

Fig. 44 shows an exceptional state at a position (I 1 , 

20 J' ) . 

In the specific example shown in Fig . 44 , when 
positions (ig - 0.5, jg - 0.5) are converted using 
conversion parameters (ak 1 , bk ' , ck ' , dk 1 , Tkx' , Tky' ) , 
converted points (converted positions) Gil' and G15 1 are 
25 located at positions where x = I' - 1 is satisfied, and 
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converted points G12 1 and G16 ' are located at positions 
where x = 1 1 + 1 is satisfied. 

Furthermore, converted points G13 ' and G14 1 are located 
at positions where y = J' - 1 is satisfied, and converted 
5 points G17 1 and G19 ■ are located at positions where y = J 1 + 
1 is satisfied. 

In the specific example shown in Fig. 44, converted 
positions (x, y) falling within the region in the coordinate 
system defined by I 1 - 2 < x < I' +2 and J ' - 2 < y < J* + 
10 2 have a special relation with the position (I', J'), that 
is, x = I 1 ± 1 or y = J' ±1 for all combinations of 
integers (k, ig, jg) . In this state, there is no G signal 
data (pixel value Gobs(k, ig, jg) ) in a region (I* ±1, J' ± 
1) (a square region with a size of 2 x 2) centered at the 
15 position (I ' , J 1 ) . 

In such a state (exceptional state) , if the true green 
light intensity Lg(I' , J ! ) at the position (I', J ! ) is 
determined in accordance with equation (46) , the resultant 
true green light intensity Lg(I', J') is unreliable 
20 (unstable), as described above. 

To avoid the above problem, when a pixel at the 
position (I 1 , J ! ) is in such an exceptional state, the 
calculation circuit 24 performs an exceptional process. 
More specifically, for a given position (I', J 1 ) of 
25 interest, when the true green light intensity Lgd 1 , J*) of 
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a pixel of an output image at the position (I 1 , J') of 
interest is estimated (hereinafter, such a pixel will be 
referred to simply as a pixel of interest) , if the pixel of 
the interest (the position (I", J') of the pixel of 
5 interest) is in the exceptional state, the calculation 

circuit 24 determines the true green light intensity Lg(I', 
J') at the position (I 1 , J') of interest of the pixel of the 
interest in the exceptional state by using pixel values 
Gobs(k, ig, jg) of the k-th captured image whose corrected 

10 position (x, y) is located in the vicinity of the position 
of a pixel of the output image located in the vicinity of 
the pixel of interest, in addition to pixel values Gobs(k, 
i^/ jg) of the k-th captured image whose corrected position 
(x, y) is located in the vicinity of the position (I 1 , J') 

15 of interest. As for the pixel (adjacent pixel) located 

adjacent to the pixel of interest located at the position 
(I 1 , J 1 ) of interest, pixels at positions (I' - 1, J'), (I 1 
+1, J 1 ) / (I', J ' -1), and (I', J' + 1) shown in Fig. 45 
may be employed. 

20 As described earlier with reference to Fig. 3, the 

imaging device 4 of the digital camera 1 according to the 
present embodiment has a Bayer-array structure. In the 
Bayer-array structure, pixels for sensing the green light 
component are disposed every two pixel positions in both X 

25 and Y direction (Fig. 3) . In this eighth embodiment, it is 
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assumed that the pixels of the imaging device 4 arranged 
such that W pixels are arranged in a horizontal (X) 
direction and H pixels are arranged in a vertical (Y) 
direction, and thus the imaging device 4 has a total of W x 
5 H pixels . 

When there is no observed value (pixel value) Gobs(k, 
jg) of the G signal in the vicinity of the position (I 1 , 
J 1 ) of interest, the pixel of interest at the position (I 1 , 
J') of interest cannot be a pixel for sensing the green 

10 light component in the Bayer-array structure. 

For example, as shown in Fig. 46, when the pixel of 
interest at the position (I 1 , J') of interest is a pixel B12 
denoted by an open circle for sensing the blue light 
component in the Bayer-array structure, there is a pixel for 

15 sensing the green light component at an immediately adjacent 
location to the left or right of or above or below the pixel 
B12 . Similarly, when the pixel of interest is a pixel of 
the imaging device 4 for sensing the red or blue light 
component, there is a pixel for sensing the green light 

20 component at an immediately adjacent location to the left or 
right of or above or below the pixel of interest. 

Therefore, when there is no observed value Gobs(k, ig, 
jg) of the G signal in the vicinity of the position (I 1 , J') 
of interest (when there is no green pixel of the captured 

25 image whose converted position (x, y) is located in the 
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vicinity of the position (I', J') of interest), there is an 
observed value of the G signal in the vicinity of one of 
positions (I', J ' - 1), (I*, J ' + 1), (I' - 1, J"), and (I' 
+ 1, J 1 ) of adjacent pixels at immediately adjacent 
5 locations above or below or to the left or right of the 
pixel of interest at the position (I', J 1 ) of interest. 
When there is an observed value Gobs(k, ig, jg) of the G 
signal in the vicinity of particular one of adjacent pixels 
located at positions (I 1 - 1, J'), (I' + 1, J 1 ), (I', J 1 - 

10 1), and (I 1 , J' + 1), the particular one of adjacent pixels 
is not in the exceptional state. For this particular 
adjacent pixel located at position (I' - 1, J ! ) , (I 1 + 1 # 
J'), (I r , J' - 1), or (I 1 , J 1 +1), a reliable value of the 
green light intensity Lg(r , J') can be determined using 

15 equation (46) . 

As described above, when the pixel of interest is in 
the exceptional state, the calculation circuit 24 determines 
the true green light intensity Lg(I', J') at the position 
(I', J') of interest by using pixel values Gobs(k, ig, jg) 

20 of pixels whose corrected position (x, y) is located in the 
vicinity of an adjacent pixel located at (I* - 1, J 1 ), (I' + 
1, J'), (I 1 , J 1 - 1), or (I", J' +1) adjacent to the pixel 
of interest, in addition to pixel values Gobs(k, ig, jg) of 
pixels whose corrected position (x, y) is located in the 

25 vicinity of the position (!', J 1 ) of interest. 
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More particularly, in this case, the calculation 
circuit 24 determines the true green light intensity Lg(I', 
J 1 ) at the position (I 1 , J') of interest by performing the 
exceptional process in accordance with equation (56) . 

Lg ( I ' , J 1 ) = 

{numerator value of G signal at (I 1 , J 1 ) 

+ numerator value of G signal at (I'- 1, J') 

+ numerator value of G signal at (I' + 1, J') 

+ numerator value of G signal at (I', J 1 - 1) 

+ numerator value of G signal at (I 1 , J' + 1)}/ 

{denominator value of G signal at (I', J') 

+ denominator value of G signal at (I 1 - 1, J 1 ) 

+ denominator value of G signal at (I' + 1, J 1 ) 

+ denominator value of G signal at (I', J' - 1) 

+ denominator value of G signal at (I 1 , J'+ 1)} 

• • • (56) 

Equation (56) represents that the true green light 
intensity Lg(I', J 1 ) is obtained by dividing the sum of the 
values of the numerator of equation (46) obtained in the 
normal process for total of five positions, that is, the 
position (I', J') of interest of the pixel of interest and 
the positions (I' - 1, J'), (I 1 + 1, J 1 ), (I 1 , J 1 - 1), and 
(I', J' + 1) of pixels adjacent to the pixel of interest by 
the sum of the values of the denominator of equation (46) 
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obtained in the normal process for total of five positions, 
that is, the position (I 1 , J') of interest of the pixel of 
interest and the positions I' - 1, J'), (I 1 + 1, J 1 ), (I', 
J' - 1), and (I 1 , J' + 1) of pixels adjacent to the pixel of 
5 interest, that is, by dividing the sum of the values of 
equation (47) by the sum of the values of equation (48) . 

Note that when the pixel of interest is in the 
exceptional state, the equation used to determine the true 
green light intensity Lg(I ! , J') at the position (I 1 , J') of 
10 interest is not limited to equation (56) but other equations 
may be used. For example, equation (57) shown below may be 
used for this purpose. 



LG(I', J) = 



15 



{(numerator value of G signal at (I 1 - 1, J 1 ))/ 



(denominator value of G signal at (I* - 1, J')) 



+ (numerator value of G signal at (I' + 1, J 1 ))/ 



(denominator value of G signal at (I' + 1, J')) 



+ (numerator value of G signal at (I 1 , J' - 1) ) / 



20 



(denominator value of G signal at (I 1 , J 1 - 1) ) 



+ (numerator value of G signal at (I', J' + 1))/ 



(denominator value of G signal at (I 



• , J' + 1) ) }/4 



• • ■ (57) 



25 



Equation (57) represent that the estimated true green 



- 239 - 

S04P0831 

light intensity Lgd 1 , J') at the (I 1 , J') of interest is 
obtained by calculating the mean value of true green light 
intensities Lg(I ! - 1, J'), Lg(I' + 1, J'), Lg(I», J ' - 1) 
and Lg(I', J 1 + 1) calculated using equation (46) for four 
5 positions (I 1 - 1, J'), (I» +1, J'), (I 1 , J' - 1), and (I', 
J' +1) of pixels adjacent to the pixel of interest. 

The weighted sum associated with the red light 
intensity given by equation (50) will be further discussed 
below. 

10 As with the case in which the true green light 

intensity Lgd', J') is calculated according to equation 
(46), when the true red light intensity Lr(I', J 1 ) is 
calculated according to equation (50) , the solution becomes 
unstable if the pixel of interest is in the exceptional 

15 state. More specifically, for a given position (I', J') of 
interest, when positions (ir - 0.5, jr - 0.5) are converted 
into the reference coordinate system using conversion 
parameters (ak 1 , bk ' , ck ' , dk 1 , Tkx', Tky ' ) , there is a 
possibility that all converted positions (x, y) of pixel 

20 values Robs(k, ir, jr) appearing within the range defined by 
I' -2<x<I'+2 and J 1 -2<y<J'+2 satisfy x = I' ± 
1 or y - J' ± 1 as shown in Fig. 44. In such a state 
(exceptional state) , there is no R signal data (pixel value 
Robs(k, ir, jr)) in a region (I 1 ±1, J' ±1) (a square 

25 region with a size of 2 x 2) centered at the position (I 1 , 
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J' ) . 

In this case, the calculation circuit 24 performs an 
exceptional process as described below. 

That is, the calculation circuit 24 determines the true 
red light intensity Lr(I', J') at the position (I', J ! ) of 
interest of the pixel of the interest in the exceptional 
state by using pixel values Robs(k, ir, jr) of the k-th 
captured image whose corrected position is located in the 
vicinity of the position of a pixel of the output image 
located in the vicinity of the pixel of interest, in 
addition to pixel values Gobs(k, ir, jr) of the k-th 
captured image whose corrected position is located in the 
vicinity of the pixel of interest. As for the pixel 
(adjacent pixel) located adjacent to the pixel of interest 
located at the position (I 1 , J ! ) of interest, pixels at 
positions I' - 1, J' - 1), (I 1 , J 1 -1), (I' + 1, J' - 1), 
(I' - 1, J'), (I« ■+ 1, J'), (I' - 1, J' + 1), (I 1 , J' + 1), 
and (I' + 1, J ' + 1) shown in Fig. 4 7 may be employed. 

When there is no observed value (pixel value) Robs(k, 
ir, jr) of the R signal in the vicinity of the position (I', 
J 1 ) of interest, the pixel of interest at the position (I 1 , 
J ! ) of interest cannot be a pixel for sensing the red light 
component in the Bayer-array structure. 

For example, as shown in Fig. 48, when the pixel of 
interest at the position (I 1 , J') of interest is a pixel Gil 
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denoted by an open circle for sensing the green light 
component in the Bayer-array structure, there is a pixel for 
sensing the red light component at an immediately adjacent 
location above or below the pixel Gil. 
5 On the other hand, for example, as shown in Fig. 48, 

when the pixel of interest at the position (I 1 , J') of 
interest is a pixel G22 denoted by an open circle for 
sensing the green light component in the Bayer-array 
structure, there is a pixel for sensing the red light 

10 component at an immediately adjacent location to the left or 
right of the pixel G22 . 

Furthermore, for example, as shown in Fig. 48, when the 
pixel of interest at the position (I 1 , J 1 ) of interest is a 
pixel B14 denoted by an open circle for sensing the blue 

15 light component in the Bayer-array structure, there is a 

pixel for sensing the red light component at an immediately 
adjacent location to the upper right, lower right, upper 
left, or lower left of the pixel B14 . 

Similarly, when the pixel of interest is a pixel other 

20 than pixels for sensing the red light component, there is a 
pixel for sensing the red light component at an immediately 
adjacent location to the left, right, upper right, lower 
right, upper left, lower left of the pixel B14 or above or 
below the pixel B14 . 

25 Therefore, when there is no observed value Robs(k, ir, 
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jr) of the R signal in the vicinity of the position (I 1 , J') 
of interest (when there is no red pixel of the captured 
image whose converted position (x, y) is located in the 
vicinity of the position (I 1 , J') of interest), there is an 
5 observed value of the R signal in the vicinity of one of 

positions (I' - 1, J 1 -1), (I', J' -1), (I 1 + 1, J f -1), 
(I' - 1, J'), (I" + l, J 1 ) , (I 1 - 1 , J 1 + 1), (I', J' + 1), 
and (I* +1, J ' +1) of adjacent pixels at immediately 
adjacent locations above or below or to the left, right, 

10 upper right, lower right, upper left, lower left of the 
pixel of interest at the position (I', J') of interest. 
When there is an observed value Robs(k, ir, jr) of the R 
signal in the vicinity of particular one of adjacent pixels 
located at positions (I 1 - 1, J 1 -1), (I 1 , J ' -1), (I' + 1, 

15 J 1 - 1), (I' - 1, J'), (I' + 1, J'), (I' - 1 , J 1 + 1), (I', 
J' +1), and (I 1 + 1, J 1 + 1), the particular one of 
adjacent pixels is not in the exceptional state. For this 
particular adjacent pixel located at position (I' - 1, J 1 - 
1), (I', J 1 - 1), (I> + 1, J» - 1), (I" - 1, J 1 ), (I 1 + 1, 

20 J'), (I« - 1 , J ' + 1), (I', J 1 + 1), or (I« + 1 , J 1 + 1), a 
reliable value of the red light intensity Lr ( I ' , J') can be 
determined using equation (50) . 

As described above, when the pixel of interest is in 
the exceptional state, the calculation circuit 24 determines 

25 the true red light intensity Lr(I', J') at the position (I ! , 
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J ! ) of interest by using pixel values Robs(k, ir, jr) of 
pixels whose corrected position (x, y) is located in the 
vicinity of an adjacent pixel located at (I* - 1, J' -1), 
( 1 1 / J' - 1), (I* + 1, J' - 1), (I' - 1/ J'), <I ! + 1, J'), 
5 (I 1 - 1, J 1 +1), (I 1 , J 1 +1), or (I« +1, J' +1) adjacent 
to the pixel of interest, in addition to pixel values Robs(k, 
ir, jr) of pixels whose corrected position (x, y) is located 
in the vicinity of the position (I', J') of interest. 
More particularly, in this case, the calculation 
10 circuit 24 determines the true red light intensity Lr(I', 
J') at the position (I', J') of interest by performing the 
exceptional process in accordance with equation (58) . 
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25 + denominator value of R signal at (I 1 - 1, J' - 1) 
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+ denominator value of R signal at (I', J' -1) 
+ denominator value of R signal at (I* +1, J' -1) 
+ denominator value of R signal at (I* - 1, J') 
+ denominator value of R signal at (I' + 1, J') 
5 + denominator value of R signal at (I 1 -1, J' +1) 

+ denominator value of R signal at (I', J' + 1) 
+ denominator value of R signal at (I f + 1, J' + 1)) 

• • • (58) 

Equation (58) represents that the true red light 

10 intensity Lr(I' # J 1 ) is obtained by dividing the sum of the 
values of the numerator of equation (50) obtained in the 
normal process for total of nine positions, that is, the 
position (I 1 , J') of interest of the pixel of interest and 
the positions (I» - 1, J' - 1), (I', J 1 -1), (I' + 1, J* - 

15 1), (I' - 1, J«), (I' + 1, J'), (I' - 1 # J 1 + 1), (I', J ' + 
1), and (I 1 + 1, J f + 1) of pixels adjacent to the pixel of 
interest by the sum of the values of the denominator of 
equation (50) obtained in the normal process for total of 
nine positions, that is, the position (I*, J') of interest 

20 of the pixel of interest and the positions (I 1 - 1, J' -1), 
J ' - 1), (I ! + 1 , J 1 - 1), (I' - 1, J'), (I' + 1, J'), 
(I' -1, J» +1), (I', J f +1), and (I 1 +1, J ' +1) of 
pixels adjacent to the pixel of interest, that is, by 
dividing the sum of the values of equation (52) by the sum 

25 of the values of equation (53) . 
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As with equation (57) associated with the G signal, 
instead of using equation (58) , the true red light intensity 
Lr(r # J 1 ) at the position (I 1 , J ! ) of interest may be 
determined by calculating the mean value of true red light 
5 intensities Lr ( I ' - 1, J' - 1) , Lr(I', J ' - 1), Lr(I' + 1, 
J' - 1), Lr(I' - 1, J'), Lr(r + 1, J'), Lr(I' - 1, J ' + 1), 
Lr(I ! , J' + 1), and Lr(I' + 1, J ' + 1) of pixels at eight 
positions (I 1 - 1, J ' - 1), (I', J 1 - 1), (I' +1, J 1 - 1), 
(I- - 1, J'), ( 1 1 + 1, J'), (I' - 1, J' + 1), (I 1 , J 1 + 1), 

10 and ( I ' + 1 , J 1 + 1) adjacent to pixel of interest. 

When the true blue light intensity Lb ( I 1 , J') is 
determined using the weighted sum equation (51) associated 
with the blue light intensity, as with the true green light 
intensity Lg(I f , J') determined using the weighted sum 

15 equation (46) associated with the green light intensity or 
as with the true red light intensity Lr(I', J') determined 
using the weighted sum equation (50) associated with the red 
light intensity, the solution becomes unstable if the pixel 
of interest at the position of interest is in the 

20 exceptional state. 

In this case, the calculation circuit 24 performs an 
exceptional process as described below. 

In the Bayer-array structure, pixels for sensing the 
blue light component are arranged in a similar manner in 

25 terms of positional relationship to the manner in which 
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pixels for sensing the red light component are arranged. 
Therefore, the calculation circuit 24 determines the true 
blue light intensity Lbd 1 , J') at the position J') of 

interest of the pixel of the interest in the exceptional 
5 state according to equation (59) similar to equation (58) . 
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• ■ ■ (59) 

Equation (59) represents that the true blue light 
intensity Lb (I', J') is obtained by dividing the sum of the 
values of the numerator of equation (51) obtained in the 
5 normal process for total of nine positions, that is, the 
position (I 1 , J') of interest of the pixel of interest and 
the positions (I» - 1, J 1 - 1) , (I', J' -1), (I 1 + 1 , J ' - 
1), (I - 1, J'), (I' + 1, J'), (I' - 1 , J 1 + 1), (I', J ' + 
1), and (I' +1, J 1 +1) of pixels adjacent to the pixel of 

10 interest the sum of the values of the denominator of 

equation (51) obtained in the normal process for total of 
nine positions, that is, the position (I 1 , J') of interest 
of the pixel of interest and the positions (I' - 1, J ' - 1), 
(I', J' - 1), (I' + 1 , J 1 - 1), (I - 1, J'), (I 1 + 1, J'), 

15 (I'-l, J'+l), (I», J» + 1) , and (I 1 + 1 , J 1 + 1) of 
pixels adjacent to the pixel of interest, that is, by 
dividing the sum of the values of equation (54) by the sum 
of the values of equation (55) . 

As with equation (57) associated with the G signal, 

20 instead of using equation (59) , the true blue light 

intensity Lb(I', J f ) at the position (I', J 1 ) of interest 
may be determined by calculating the mean value of true blue 
light intensities Lb (I 1 - 1, J' - 1), Lb (I', J' - 1), Lb ( I ' 
+ 1, J' - 1), Lb ( 1 1 - 1, J'), Lb ( I 1 + 1, J'), Lb ( I 1 - 1, J* 

25 +1), Lb(I', J' +1), and Lb(I f + 1, J' + 1) of pixels at 
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eight positions (I>- 1, J' -1), (I', J 1 -1), (I 1 + 1, J 1 - 
1), (I" - 1, J'), (I" + 1, J 1 ) # (I" - 1 / J 1 + 1), (I', J' + 
1), and (I' + 1, J' + 1) adjacent to pixel of interest. 

Now, referring to flow charts shown in Figs. 4 9 and 50, 
5 a correct image estimation process in step S4 shown in Fig. 
2 according to an eighth embodiment of the present invention 
is described. 

First, in step S201, the calculation circuit 24 sets a 
variable J 1 of (I', J 1 ) to 1 to indicate the position in the 

10 Y direction of a pixel in the reference coordinate system. 
Thereafter, the process proceeds to step S202. 

In step S202, the calculation circuit 24 calculates 
sets a variable I 1 of (I 1 , J') to 1 to indicate the position 
in the X direction of a pixel in the reference coordinate 

15 system. Thereafter, the process proceeds to step S203 . 

Note that variables I' and J' are also used to indicate the 
position in the X and Y directions of a pixel of an output 
image . 

In step S203, the calculation circuit 24 selects the 
20 position (I 1 , J') as a position of interest, and, for the 
position (I 1 , J 1 ) of interest, the calculation circuit 24 
determines all combinations of integers (k, ig, jg) that 
satisfy I 1 - 2 < x < I 1 + 2 and J 1 - 2 < y < J' + 2 with 
respect to the converted position (x, y) in the reference 
25 coordinate system obtained by converting, based on 
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conversion parameters (ak», bk 1 , ck 1 , dk 1 , Tkx ' , Tky ' ) , the 
center position (ig - 0.5, jg - 0.5) of the pixel that 
senses the green component of the k-th image, for all of 
first to N-th images. Thereafter, the process proceeds to 
5 step S204. 

In step S204, the calculation circuit 24 calculates the 
denominator of the weighted sum equation associated with the 
green light intensity according to equation (48) and the 
numerator of the weighted sum equation associated with the 

10 green light intensity according to equation (47) by using 

all combinations of integers (k, ig, jg) determined in step 
S203. The calculation circuit 24 stores the result of the 
calculation in a memory (not shown) . Thereafter, the 
process proceeds to step S205. 

15 In step S205, for the position (I', J 1 ) of interest, 

the calculation circuit 24 determines all combinations of 
integers (k, ir, jr) that satisfy I 1 -2<x<I ! +2 and J ' 
- 2 < y < J' +2 with respect to the converted position (x, 
y) in the reference coordinate system obtained by converting, 

20 based on conversion parameters (ak 1 , bk ' , ck ' , dk ' , Tkx', 

Tky 1 ), the center position (ir - 0.5, jr - 0.5) of the pixel 
that senses the red component of the k-th image, for all of 
first to N-th images. Thereafter, the process proceeds to 
step S206. 

25 In step S206, the calculation circuit 24 calculates the 
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denominator of the weighted sum equation associated with the 
red light intensity according to equation (53) and the 
numerator of the weighted sum equation associated with the 
red light intensity according to equation (52) by using all 
5 combinations of integers (k, ir, jr) determined in step S205. 
The calculation circuit 24 stores the result of the 
calculation in the memory. Thereafter, the process proceeds 
to step S207. 

In step S207, for the position (I 1 , J ! ) of interest, 

10 the calculation circuit 24 determines all combinations of 

integers (k, ib, jb) that satisfy I 1 -2<x<I ! +2 and J 1 
-2<y<J' +2 with respect to the converted position (x, 
y) in the reference coordinate system obtained by converting, 
based on conversion parameters (ak', bk 1 , ck ' , dk 1 , Tkx ' , 

15 Tky 1 ), the center position (ib - 0.5, jb - 0.5) of the pixel 
that senses the blue component of the k-th image, for all of 
first to N-th images. 

In step S208, the calculation circuit 24 calculates the 
denominator of the weighted sum equation associated with the 

20 blue light intensity according to equation (55) and the 

numerator of the weighted sum equation associated with the 
blue light intensity according to equation (54) by using all 
combinations of integers (k, ib, jb) determined in step S207. 
The calculation circuit 24 stores the result of the 

25 calculation in the memory. Thereafter, the process proceeds 
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to step S209. 

In step S209, the calculation circuit 24 determines 
whether the variable 1 1 is equal to the number of pixels W 
in the X direction. If it is determined in step S209 that 
5 the variable 1 1 is not equal to the number of pixel W, that 
is, if steps S203 to S208 are not completed for all pixels 
whose Y position is J', the process proceeds to step S210. 
In step S210, the calculation circuit 24 increments the 
variable I* by 1. Thereafter, the process returns to step 
10 S203. 

In the case in which it is determined in step S209 that 
the variable I' is equal to the number of pixel W, that is, 
if steps S203 to S208 are completed for all pixels whose Y 
position is J', the process proceeds to step S211. 

15 In step S211, the calculation circuit 24 determines 

whether the variable J 1 is equal to the number of pixels H 
in the Y direction. If it is determined in step S211 that 
the variable J f is not equal to the number of pixel H, that 
is, if steps S203 to S208 are not completed for all Y 

20 positions of the imaging device 4, the process proceeds to 
step S212. In step S212, the calculation circuit 24 
increments the variable J' by 1 . Thereafter, the process 
returns to step S202. 

On the other hand, if it is determined in step S211 

25 that the variable J' is equal to the number of pixel H, that 
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is, if steps S203 to S208 are completed for all Y positions 
of the imaging device 4, the process proceeds to step S213 
in Fig. 50. Note that the numbers of pixels W and H are 
equal to the numbers of pixels of the output image in the X 
5 and Y directions, respectively. 

In step S213, as in step S201, the calculation circuit 
24 sets the variable J 1 to 1 . Thereafter, the process 
proceeds to step S214 . 

In step S214, ass in step S202, the calculation circuit 

10 24 sets the variable I 1 to 1 . Thereafter, the process 
proceeds to step S215. 

In step S215, the calculation circuit 24 calculates 
employs the position (I 1 , J') as the position of interest, 
and calculates the true green light intensity Lg(I ! , J') at 

15 the position (I', J') of interest. Thereafter, the process 
proceeds to step S216. More specifically, in step S215 
described above, the true green light intensity Lg ( I ■ , J') 
at the position (I', J') of interest is determined by 
performing either the normal process (first calculation 

20 process) using the weighted sum equation (46) associated 
with the green light intensity or the exceptional process 
(second calculation process) according to equation (56) , as 
will be described in further detail later. 

In step S216, the calculation circuit 24 calculates 

25 employs the position (I', J') as the position of interest, 
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and calculates the true red light intensity Lr(I', J') at 
the position (I 1 , J ! ) of interest. Thereafter, the process 
proceeds to step S217. More specifically, in step S216 
described above, the true red light intensity Lr(I', J 1 ) at 
5 the position (I', J f ) of interest is determined by 

performing either the normal process (first calculation 
process) using the weighted sum equation (50) associated 
with the red light intensity or the exceptional process 
(second calculation process) according to equation (58) , as 

10 will be described in further detail later. 

In step S217, the calculation circuit 24 employs the 
position (I 1 , J 1 ) as the position of interest, and 
calculates the true blue light intensity Lb (I 1 , J') at the 
position (I', J') of interest. Thereafter, the process 

15 proceeds to step S218. More specifically, in step S217 

described above, the true blue light intensity Lb ( I 1 , J') at 
the position (I 1 , J 1 ) of interest is determined by 
performing either the normal process (first calculation 
process) using the weighted sum equation (51) associated 

20 with the blue light intensity or the exceptional process 

(second calculation process) according to equation (59) , as 
will be described in further detail later. 

In step S218, the calculation circuit 24 determines 
whether the variable I ' is equal to the number W of pixels 

25 in the X direction. If it is determined in step S218 that 
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the variable I' is not equal to the number of pixel W, that 
is, if steps S215 to S217 are not completed for all pixels 
whose Y position is J ! , the process proceeds to step S219. 
In step S219, the calculation circuit 24 increments the 
5 variable 1 1 by 1. Thereafter, the process returns to step 
S215. 

In the case in which it is determined in step S218 that 
the variable I 1 is equal to the number of pixel W, that is, 
if steps S215 to S217 are completed for all pixels whose Y 

10 position is J', the process proceeds to step S220. 

In step S220, the calculation circuit 24 determines 
whether the variable J 1 is equal to the number of pixels H 
in the Y direction. If it is determined in step S220 that 
the variable J 1 is not equal to the number of pixel H, that 

15 is, if steps S215 to S217 are not completed for all Y 

positions of the imaging device 4, the process proceeds to 
step S221. In step S221, the calculation circuit 24 
increments the variable J' by 1. Thereafter, the process 
returns to step S214. 

20 On the other hand, if it is determined in step S220 

that the variable J ! is equal to the number of pixel H, that 
is, if steps S215 to S217 are completed for all Y positions 
of the imaging device 4, the process proceeds to step S222 . 
In step S222, the calculation circuit 24 estimates an 

25 output image (signal) from the true green light intensity 
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Lg(I', J 1 ), the true red light intensity Lr(I>, J'), and the 
true blue light intensity Lb ( I 1 , J') determined respectively 
steps S215, S216, and S217. The resultant output image is 
supplied to the D/A converter 9 or the codec 12. Thereafter, 
5 the processing flow returns from the current process. More 
specifically, for the (i-th, j-th) pixel, the calculation 
circuit 24 estimates a correct image signal of the (i-th, j- 
th) pixel from the true green light intensity Lg (= Lg(I', 
J')) determined as the green value (G signal) in step S215, 

10 the true red light intensity Lr(i - 0.5, j - 0.5) (= Lr(I', 
J')) determined as the red value (R signal) in step S216, 
and the true blue light intensity Lb(i - 0.5, j - 0.5) (= 
Lb(I', J 1 )) determined as the blue value (B signal) in step 
S217. The calculation circuit 24 estimates an output image 

15 by performing the above-described process for all pixels of 

the output image whose center is located at the position (I 1 , 
J' ) (= (i - 0.5, j - 0.5) ) . 

Now referring to a flow chart shown in Fig. 51, the 
process performed in step S215 in Fig. 50 to calculate the 

20 true green light intensity Lg(I' , J') at the position (I 1 , 
J 1 ) of interest is described below. 

First, in step S251, the calculation circuit 24 
determines whether the absolute value of the denominator of 
the weighted sum of green light intensities in equation (46) 

25 calculated in step S204 in Fig. 49 for the position (I 1 , J') 
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of interest, that is, the absolute value of equation (48) , 
is equal to or greater than a threshold value. The 
threshold value is used to determine whether the absolute 
value of equation (48) should be regarded as being equal to 
5 0 and thus the exception handling process should be 

performed. The threshold value is set in advance in the 
calculation circuit 24 . The threshold value may be set 
variably to a value specified by a user. 

If it is determined in step S251 that the absolute 

10 value of equation (48) for the position (I', J') of interest 
is equal to or greater than the threshold value, that is, if 
the absolute value of equation (48) for the position (I 1 , 
J') of interest is not so small that it can be regarded as 
equal to 0, the process proceeds to step S252. In step S252, 

15 the calculation circuit 24 selects the normal process and 
calculates the weighted sum equation (46) associated with 
the green light intensity. That is, the calculation circuit 
24 divides the value of the numerator of the weighted sum 
equation (46) associated with the green light intensity 

20 calculated in step S204, that is, the value of equation (47) 
by the value of the denominator of the weighted sum equation 
(46) associated with the green light intensity calculated in 
step S204, that is, the value of equation (48) . Thus, in 
step S252, the true green light intensity L»g ( I 1 , J') at the 

25 position (I 1 , J') of interest is determined. 
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On the other hand, if it is determined in step S251 
that the absolute value of equation (48) for the position 
(I 1 , J') of interest is less than the threshold value, that 
is, the absolute value of equation (48) is equal to or 
5 nearly equal to 0, the process proceeds to step S253 . In 
step S253, the calculation circuit 24 selects the 
exceptional process and performs it. That is, the 
calculation circuit 24 determines the light intensity Lg ( I ' , 
J') of the true green color at a position (I', J') of 

10 interest by calculating equation (56) . 

Now referring to a flow chart shown in Fig. 52, the 
process performed in step S216 in Fig. 50 to calculate the 
true red light intensity Lr(I», J') at the position (I 1 , J') 
of interest is described below. 

15 First, in step S271, and the calculation circuit 24 

determines whether the absolute value of the denominator of 
the weighted sum of red light intensities in equation (50) 
calculated in step S206 in Fig. 49 for the position (I 1 , J') 
of interest, that is, the absolute value of equation (53), 

20 is equal to or greater than a threshold value. The 

threshold value is used to determine whether the absolute 
value of equation (53) should be regarded as being equal to 
0 and thus the exception handling process should be 
performed. The threshold value is set in advance in the 

25 calculation circuit 24. The threshold value may be set 
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variably to a value specified by a user. Note that the 
threshold value may or may not be equal to the threshold 
value used in step S251 in Fig. 51. 

If it is determined in step S271 that the absolute 
5 value of equation (53) for the position (I 1 , J') of interest 
is equal to or greater than the threshold value, that is, if 
the absolute value of equation (53) for the position (I', 
J') of interest is not so small that it can be regarded as 
equal to 0, the process proceeds to step S272 . In step S272, 

10 the calculation circuit 24 selects the normal process and 
calculates the weighted sum equation (50) associated with 
the red light intensity. That is, the calculation circuit 
24 divides the value of the numerator of the weighted sum 
equation (50) associated with the red light intensity 

15 calculated in step S206, that is, the value of equation (52) 
by the value of the denominator of the weighted sum equation 
(50) associated with the red light intensity calculated in 
step S206, that is, the value of equation (53). Thus, in 
step S272, the true red light intensity Lr(I', J 1 ) at the 

20 position (I 1 , J 1 ) of interest is determined. 

On the other hand, if it is determined in step S271 
that the absolute value of equation (53) for the position 
(I', J') of interest is less than the threshold value, that 
is, the absolute value of equation (53) is equal to or 

25 nearly equal to 0, the process proceeds to step S273 . In 
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step S273, the calculation circuit 24 selects the 
exceptional process and performs it. That is, the 
calculation circuit 24 determines the light intensity Lr(I', 
J') of the true red color at a position (I 1 , J') of interest 
5 by calculating equation (58) . 

Now referring to a flow chart shown in Fig. 53, the 
process performed in step S217 in Fig. 50 to calculate the 
true blue light intensity Lg(I ! , J') at the position (I', 
J') of interest is described below. 

10 First, in step S291, and the calculation circuit 24 

determines whether the absolute value of the denominator of 
the weighted sum of blue light intensities in equation (49) 
calculated in step S208 in Fig. 49 for the position (I', J') 
of interest, that is, the absolute value of equation (55), 

15 is equal to or greater than a threshold value. The 

threshold value is used to determine whether the absolute 
value of equation (55) should be regarded as being equal to 
0 and thus the exception handling process should be 
performed. The threshold value is set in advance in the 

20 calculation circuit 24. The threshold value may be set 
variably to a value specified by a user. Note that the 
threshold value may or may not be equal to the threshold 
value used in step S251 in Fig. 51 or that used in step S271 
in Fig. 52. 

25 If it is determined in step S291 that the absolute 
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value of equation (55) for the position (I 1 , J') of interest 
is equal to or greater than the threshold value, that is, if 
the absolute value of equation (55) for the position (I 1 , 
J ! ) of interest is not so small that it can be regarded as 
5 equal to 0, the process proceeds to step S292. In step S292, 
the calculation circuit 24 selects the normal process and 
calculates the weighted sum equation (51) associated with 
the blue light intensity. That is, the calculation circuit 
24 divides the value of the numerator of the weighted sum 

10 equation (51) associated with the blue light intensity- 
calculated in step S208, that is, the value of equation (54) 
by the value of the denominator of the weighted sum equation 
(51) associated with the blue light intensity calculated in 
step S208, that is, the value of equation (55) . Thus, in 

15 step S292, the true blue light intensity Lb(I', J') at the 
position (I', J ! ) of interest is determined. 

On the other hand, if it is determined in step S291 
that the absolute value of equation (55) for the position 
(I 1 , J') of interest is less than the threshold value, that 

20 is, the absolute value of equation (55) is equal to or 

nearly equal to 0, the process proceeds to step S293. In 
step S293, the calculation circuit 24 selects the 
exceptional process and performs it. That is, the 
calculation circuit 24 determines the light intensity Lb(I', 

25 J') of the true blue color at a position (I', J') of 
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interest by calculating equation (59) . 

As described above, in the eighth embodiment, the true 
green light intensity Lg ( I ' , J'), the true red light 
intensity Lr(I', J ! ), and the true blue light intensity 
5 Lb (I', J') are determined by calculating the weighted sum 
using the cubic function having a characteristic similar to 
that of the lowpass filter as the weight depending on the 
distance between the position (I', J') of interest and the 
converted position (x, y) in the vicinity of the position 

10 (I 1 , J') of interest, thereby determining a sharp image very 
close to an ideal image. 

In the eighth embodiment, for the position J') of 

interest, when positions (ig - 0.5, jg - 0.5) are converted 
into the reference coordinate system using conversion 

15 parameters (ak ! , bk 1 , ck • , dk ' , Tkx ' , Tky'), if one or more 
converted positions (x, y) of pixel values Gobs(k, ig, jg) 
fall within the range defined by I 1 - 2 < x < I 1 + 2 and J 1 
- 2 < y < J' +2, then the weighted sum of those pixel 
values Gobs(k, ig, jg) whose converted positions (x, y) are 

20 located in the above-described range in the vicinity of the 
position (I 1 , J') of interest is calculated in accordance 
with the weighted sum equation (46) associated with the 
green light intensity, and the result is employed as the 
true green light intensity Lg(I', J') (normal process). 

25 However, if the absolute value of the denominator of 
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the weighted sum equation (46) associated with the green 
light intensity calculated for the position (I', J') of 
interest, that is, the value of equation (48) , is less than 
the predetermined small threshold value that can be regarded 
5 as substantially equal to 0, the solution of the weighted 
sum equation (46) becomes unstable. In such a case, the 
weighted sum of pixel values Gobs(k, ig, jg) whose 
conversion position (x, y) falls in the vicinity of the 
position (I 1 , J') of interest and pixel values Gobs(k, ig, 

10 jg) whose conversion position (x, y) falls in the vicinity 
of a pixel adjacent to the pixel of interest is calculated 
in accordance with the weighted sum equation (56) , and the 
result is employed as the true green light intensity Lg(l*, 
J ! ) (exceptional process). 

15 The true red light intensity Lr(r , J ! ) and the true 

blue light intensity Lb (I 1 , J 1 ) are determined in a similar 
manner . 

Thus, it is possible to obtain a good output image 
including substantially no noise. 
20 The normal process and the exceptional process will be 

further discussed below from another point of view. 

For example, for the green light intensity, in the 
normal process described above, the weighted sum is 
calculated using pixel values Gobs(k, ig, jg) of pixels of 
25 the captured image whose converted position (x, y) falls 
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within the vicinity of the position (I', J') of interest of 
the pixel of interest. On the other hand, in the 
exceptional process, the weighted sum is calculated using 
pixel values Gobs (k, ig, jg) of pixels of the captured image 
5 whose converted position (x, y) falls within the vicinity of 
the position (I 1 , J') of interest of the pixel of interest 
and also using pixel values Gobs (k, ig, jg) of pixels of the 
captured image whose converted position (x, y) falls within 
the vicinity of the position of a pixel adjacent to the 

10 pixel of interest . 

That is, in the exceptional process, the weighted sum 
is calculated using not only pixel values Gobs(k, ig, jg) of 
pixels of the captured image whose converted position (x, y) 
falls within the vicinity of the position (I', J') of the 

15 pixel of interest but also pixel values Gobs(k, ig, jg) of 

pixels of the captured image whose converted position (x, y) 
falls within the vicinity of a pixel adjacent to the pixel 
of interest. 

That is, as described above, in the normal process, the 
20 true green light intensity Lg(I', J') at the position (I', 
J') of interest is determined by calculating the weighted 
sum of pixel values Gobs(k, ig, jg) of the captured image 
observed within the region in the vicinity of the position 
(I 1 , J 1 ) of interest, that is, within the region defined by 
25 I 1 - 2 < x < I 1 + 2 and J' - 2 < y < J' + 2. In contrast, 
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in the exceptional process, the true green light intensity 
Lg ( I' , J') at the position (I 1 , J') of interest is 
determined by calculating the weighted sum of pixel values 
Gobs(k, ig, jg) of the captured image observed within the 
5 region in the vicinity of the position (I', J 1 ) of interest, 
which is greater than the vicinity region used in the normal 
process, that is, more specifically within the region 
defined by I' - 3 < x < I' + 3 and J ' -3<y<J' +3. 

In other words, when the true green light intensity 
10 Lg(I', J 1 ) at the position (I 1 , J') of interest is 

determined, the region in the vicinity of the position (I 1 , 
J 1 ) of interest is defined by I' - 3 < x < I' +3 and J' - 3 

< y < J' +3. In the normal process, the true green light 
intensity Lg ( I ' , J f ) at the position (I 1 , J') of interest is 

15 determined by calculating the weighted sum equation (46) , in 
which the weight is set to 0 for those pixel values Gobs(k, 
ig, jg) that are observed within the vicinity region defined 
above but that are outside the region defined by I' - 2 < x 

< I 1 + 2 and J 1 - 2 < y < J' + 2. On the other hand, in the 
20 exceptional process, the true green light intensity Lg ( 1 1 , 

J 1 ) at the position (I 1 , J') of interest is determined 
according to equation (56) , in which, for those pixel values 
Gobs(k, ig, jg) that are observed within the vicinity region 
defined above but that are outside the region defined by I 1 
25 -2<x<I f +2 and J' - 2 < y < J» +2, that is, for pixel 
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values Gobs(k, ig, jg) observed in the vicinity of a pixel 
adjacent to the pixel of interest, the weight is set not to 
0 but to a value given by the cubic function Cubic (z) whose 
origin is taken at the position of that adjacent pixel. 
5 That is, when the true green light intensity Lg(I ! , J') 

is determined by calculating the weighted sum, the weight is 
set differently (to a first or second weight) depending on 
whether the calculation is performed in the normal process 
or the exceptional process. 

10 Although in the eighth embodiment, the cubic function 

Cubic (z) in the form of equation (39) is used to define the 
weight depending on the distance z between a position (I', 
J') of interest and a nearby position (x, y) at which a 
pixel value Gobs(k, ig, jg) is observed, the weight may be 

15 defined by using another arbitrary function serving as a 
lowpass filter in characteristic, such as sin(z)/z. 

In the embodiments described above, the first captured 
image is used as the reference image, and second to N-th 
captured images are employed as target images. Instead of 

20 the first captured image, any one of the second to N-th 
captured images may be used as the reference image. 

Fig. 54 shows N images obtained by successive N 
capturing operations (at a high shutter speed) . In this 
specific example shown in Fig. 54, N = 8. 

25 First to eighth captured images 401i to 401 8 (k-th image 
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401 k ) are captured sequentially in time and displaced in a 
direction to upper right with passage of time due to camera 
shake . 

Fig. 55 shows eight captured images 401i to 401 8 similar 
5 to those shown in Fig. 54. 

In the case in which the signal processor 7 estimates 
an output image by using the first one of the captured 
images 401i to 401 8 as the reference image and second to 
eighth images as target images, the output image is obtained 
10 as an image formed in a coordinate system defined on the 

first image 401i employed as the reference image. In Fig. 55, 
the first image 401 X/ that is, the output image is 
represented by a bold line. 

The region of the output image represented by the bold 
15 line in Fig. 55 includes a region 411 located in the upper 
right corner and denoted by a dotted line in which all of 
eight captured images consisting of first to eighth images 
401i to 401 8 can be used in estimation of light intensity 
(pixel value) expected to be true. In this region 411, high 
20 image quality can be obtained because estimation of pixel 
values is performed using data of all of eight captured 
images consisting of first to eighth images 401i to 401 8 . 

However, in an area of the output image denoted by the 
bold line in Fig. 55 other than the region 411, only data of 
25 some of first to eighth captured images 401i to 401 8 are 
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available in estimation of the expected output image. In 
such an area, image quality in terms of sharpness becomes 
low compared with the region 411, because all data of the 
first to eighth captured images 401i to 401 8 cannot be used. 
5 As described above, in a set of captured images having 

successive displacement in the direction to upper right due 
to camera shake, the amount of data available in estimation 
of the output image decreases with position in an opposite 
direction, that is, to upper right, and the image quality in 

10 terms of sharpness decreases with respect to the quality of 
the region 411. 

Of data of the second to eighth captured images 401i to 
401 8 used as target images, data in a region 412 outside the 
output image denoted by the bold line in Fig. 55 cannot be 

15 used in estimation of the output image and is discarded 
uselessly. 

As can be understood from the above discussion, when 
the output image is estimated using the first captured image 
as the reference image, if displacements occur in a 

20 particular direction due to camera shake, degradation in 
image quality in terms of sharpness occurs in an area 
located away from the center of the output image in a 
direction opposite to the direction of displacement, 
although high image quality is obtained in an area (for 

25 example, the region 411 in Fig. 55) located away from the 
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center of the output image in the same direction as the 
direction of displacement. 

In general, when a user views an image, a central part 
of the image most attracts user's attention. Therefore, it 
5 is desirable that image quality in terms of sharpness be 
high in the central part of the image . 

In view of the above, in estimation of an output image, 
the signal processor 7 may employ, as the reference image, 
an intermediate image of N successively-captured images. 

10 More specifically, of N successively-captured images, an 
image captured at the exact or nearly exact center of the 
time during which N images were captured is employed as the 
reference image (hereinafter, such an image will be referred 
to simply as an intermediate image) , and the other captured 

15 images may be used as target images. 

For example, as shown in Fig. 56, of eight captured 
images 401i to 401 8 , the signal processor 7 employs a fourth 
captured image denoted by a bold line as the reference image 
and estimates an output image. In this case, data of all of 

20 the eight captured images consisting of first to eighth 

images 401i to 401 8 are available in a central area 421 when 
the output image is estimated. 

As described above, in estimation of an output image, 
by employing an intermediate image of a plurality of images 

25 captured sequentially in time as the reference image and 
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employing the other captured images as target images, it 
becomes possible to obtain an output image with higher 
quality in terms of sharpness in particular in the central 
area of the output image . 
5 In most cases, when a user takes a picture of a subject, 

the camera angle is set such that the subject is located at 
the center of an image (frame). Furthermore, as described 
above, when a user views an image, a central part of the 
image most attracts user's attention. From this point of 

10 view, an image whose central part is sharper than in 

peripheral parts can be regarded as a better image than an 
image that is opposite in terms of sharpness. 

In view of the above, as shown in Fig. 56, if an 
intermediate image is employed as the reference image and 

15 the other images are employed as target images, data of all 
of eight captured images consisting of first to eighth 
images 401i to 401 8 are used in estimation of a central part 
of an output image, and the resultant output image has 
higher quality in terms of sharpness in particular in the 

20 central part than that obtained when the first captured 
image is employed as the reference image. 

The frequency of camera shake generally falls within 
the range of 10 to 15 Hz. Therefore, in Fig. 56, for a set 
of eight images 401i to 401 8 successively captured at a total 

25 effective shutter speed of 1/50 sec or higher, the 
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displacement due to camera shake can be approximated by a 
linear displacement. That is, the camera shake can be 
regarded as movement in a particular direction at a constant 
speed. Therefore, for the set of eight images 401i to 401 8 
5 captured sequentially in time, the displacement due to the 
camera shake during the exposure time can be approximated by 
a linear displacement when the displacement is compensated 
for in the estimation of the output image, in which an 
intermediate image such as a fourth captured image 401 4 or a 

10 fifth captured image 401 5 is employed as the reference image 
thereby obtaining the output image having high image quality 
in terms of sharpness in particular in its central part, as 
described above with reference to Fig. 56. 

In the signal processor 7 shown in Fig. 4, when, for 

15 example, the fourth captured image 4 01 4 of the eight captured 
images 401i to 401 8 is employed as the reference image, the 
fourth captured image 401 4 employed as the reference image is 
supplied to the frame memory 22-1 and stored therein, and 
the first to third captured images 401i to 401 3 and the fifth 

20 to eighth captured images 401 5 to 401 8 are respectively 
supplied to the frame memories 22-2 to 22-8 and stored 
therein . 

In addition to high image quality in terms of sharpness 
in the central part of the output image, use of an 
25 intermediate image as the reference image provides a further 
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advantage that the signal processor 7 can be designed more 
easily . 

Herein, let us assume that the displacement due to the 
camera shake can be approximated by a linear displacement as 
5 described above, and let us also assume that when eight 
images 401i to 401 8 are captured sequentially in time, an 
image -to -image displacement of, for example, ten pixels 
occurs. In this case, if the first captured image is 
employed as the reference image, a displacement of at most 
10 70 pixels can occur, even if the exposure time of each image 
is infinitely small. Thus, the signal processor 7 has to be 
designed so that it can handle a displacement of at most 70 
pixels . 

In contrast, when an intermediate image, such as the 
15 fourth 401 4 image of the captured images 401i to 401 8 is 
employed as the reference image, the maximum possible 
displacement is 40 pixels. Therefore, it is sufficient for 
the signal processor 7 to handle a displacement of at most 
40 pixels, and thus it becomes easier to design the signal 
20 processor 7. 

The sequence of processing steps according to the 
eighth embodiment may be performed by means of hardware or 
software, as with the other embodiments. In the case in 
which the sequence of processing steps according to the 
25 eighth embodiment is performed by means of software, for 
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example, the digital camera 1 may be realized by executing a 
program on a computer such as that shown in Fig. 41, as with 
the other embodiments . 

In the present description, the steps described in the 
5 program stored in the storage medium may be performed either 
in time sequence in accordance with the order described in 
the program or in a parallel or separate fashion. 

Although in the embodiments described above, a single - 
plane sensor is used as the imaging device 4 of the digital 

10 camera 1, and an image having three colors for each pixel is 
produced from an image signal having one color signal for 
each pixel output from the imaging device 4, the imaging 
device 4 is not limited to the single-plane sensor. Other 
types of imaging device 4 capable of outputting n color 

15 signals for each pixel may be used, and, in the correct 

image estimation process, an image having (n + 1) or more 
color signals for each pixel may be produced from the image 
signal output from the imaging device 4 . 

In the embodiments described above, a first captured 

20 image or an intermediate captured image is used as the 

reference image, and a positional relationship between a 
target image and the reference image is detected. 
Alternatively, instead of the first captured image or the 
intermediate captured image, an arbitrary captured image may 

25 be used as the reference image, and a positional 
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relationship between a target image and the reference image 
may be detected. The present invention may be applied not 
only to a digital still camera but also to other types of 
cameras such as a digital video camera. 

Industrial Applicability 

The present invention makes it possible to obtain a 
sharper image. 



